Low-surface-brightness galaxies (LSBGs) -- defined as systems that are fainter than the surface-brightness limits of past wide-area surveys -- form the overwhelming majority of galaxies in the dwarf regime (M* < 10^9 MSun). Using New Horizon, a high-resolution cosmological simulation, we study the origin of LSBGs and explain why LSBGs at similar stellar mass show the large observed spread in surface brightness. New Horizon galaxies populate a well-defined locus in the surface brightness -- stellar mass plane, with a spread of ~3 mag arcsec^-2, in agreement with deep SDSS Stripe data. Galaxies with fainter surface brightnesses today are born in regions of higher dark-matter density. This results in faster gas accretion and more intense star formation at early epochs. The stronger resultant supernova feedback flattens gas profiles at a faster rate which, in turn, creates shallower stellar profiles (i.e. more diffuse systems) more rapidly. As star formation declines towards late epochs (z<1), the larger tidal perturbations and ram pressure experienced by these systems (due to their denser local environments) accelerate the divergence in surface brightness, by increasing their effective radii and reducing star formation respectively. A small minority of dwarfs depart from the main locus towards high surface brightnesses, making them detectable in past wide surveys. These systems have anomalously high star-formation rates, triggered by recent, fly-by or merger-driven starbursts. We note that objects considered extreme/anomalous at the depth of current datasets, e.g. `ultra-diffuse galaxies', actually dominate the predicted dwarf population and will be routinely visible in future surveys like LSST.