Modularity is a popular metric for quantifying the degree of community structure within a network. The distribution of the largest eigenvalue of a network's edge weight or adjacency matrix is well studied and is frequently used as a substitute for modularity when performing statistical inference. However, we show that the largest eigenvalue and modularity are asymptotically uncorrelated, which suggests the need for inference directly on modularity itself when the network size is large. To this end, we derive the asymptotic distributions of modularity in the case where the network's edge weight matrix belongs to the Gaussian Orthogonal Ensemble, and study the statistical power of the corresponding test for community structure under some alternative model. We empirically explore universality extensions of the limiting distribution and demonstrate the accuracy of these asymptotic distributions through type I error simulations. We also compare the empirical powers of the modularity based tests with some existing methods. Our method is then used to test for the presence of community structure in two real data applications.