Recently, graph neural networks (GNNs) have been successfully applied to recommender systems. In recommender systems, the user's feedback behavior on an item is usually the result of multiple factors acting at the same time. However, a long-standing challenge is how to effectively aggregate multi-order interactions in GNN. In this paper, we propose a Graph Factorization Machine (GFM) which utilizes the popular Factorization Machine to aggregate multi-order interactions from neighborhood for recommendation. Meanwhile, cross-domain recommendation has emerged as a viable method to solve the data sparsity problem in recommender systems. However, most existing cross-domain recommendation methods might fail when confronting the graph-structured data. In order to tackle the problem, we propose a general cross-domain recommendation framework which can be applied not only to the proposed GFM, but also to other GNN models. We conduct experiments on four pairs of datasets to demonstrate the superior performance of the GFM. Besides, based on general cross-domain recommendation experiments, we also demonstrate that our cross-domain framework could not only contribute to the cross-domain recommendation task with the GFM, but also be universal and expandable for various existing GNN models.