Generalization in Graph Neural Networks: Improved PACBayesian Bounds on Graph Diffusion
Abstract
Graph neural networks are widely used tools for graph prediction tasks. Motivated by their empirical performance, prior works have developed generalization bounds for graph neural networks, which scale with graph structures in terms of the maximum degree. In this paper, we present generalization bounds that instead scale with the largest singular value of the graph neural network's feature diffusion matrix. These bounds are numerically much smaller than prior bounds for realworld graphs. We also construct a lower bound of the generalization gap that matches our upper bound asymptotically. To achieve these results, we analyze a unified model that includes prior works' settings (i.e., convolutional and messagepassing networks) and new settings (i.e., graph isomorphism networks). Our key idea is to measure the stability of graph neural networks against noise perturbations using Hessians. Empirically, we find that Hessianbased measurements correlate with the observed generalization gaps of graph neural networks accurately. Optimizing noise stability properties for finetuning pretrained graph neural networks also improves test performance on several graphlevel classification tasks.
 Publication:

arXiv eprints
 Pub Date:
 February 2023
 DOI:
 10.48550/arXiv.2302.04451
 arXiv:
 arXiv:2302.04451
 Bibcode:
 2023arXiv230204451J
 Keywords:

 Computer Science  Machine Learning;
 Computer Science  Social and Information Networks;
 Mathematics  Statistics Theory;
 Statistics  Machine Learning
 EPrint:
 36 pages. Appeared in AISTATS 2023