For ground-based optical imaging with current CCD technology, the Poisson fluctuations in source and sky background photon arrivals dominate the noise budget and are readily estimated. Another component of noise, however, is the signal from the undetected population of stars and galaxies. Using injection of artificial galaxies into images, we demonstrate that the measured variance of galaxy moments (used for weak gravitational lensing measurements) in Dark Energy Survey (DES) images is significantly in excess of the Poisson predictions, by up to 30\%, and that the background sky levels are overestimated by current software. By cross-correlating distinct images of "empty" sky regions, we establish that there is a significant image noise contribution from undetected static sources (US), which on average are mildly resolved at DES resolution. Treating these US as a stationary noise source, we compute a correction to the moment covariance matrix expected from Poisson noise. The corrected covariance matrix matches the moment variances measured on the injected DES images to within 5\%. Thus we have an empirical method to statistically account for US in weak lensing measurements, rather than requiring extremely deep sky simulations. We also find that local sky determinations can remove the bias in flux measurements, at a small penalty in additional, but quantifiable, noise.