Motivated by the goal to give the simplest possible microscopic foundation for a broad class of topological phases, we study quantum mechanical lattice models where the topology of the lattice is one of the dynamical variables. However, a fluctuating geometry can remove the separation between the system size and the range of local interactions, which is important for topological protection and ultimately the stability of a topological phase. In particular, it can open the door to a pathology, which has been studied in the context of quantum gravity and goes by the name of 'baby universe', here we discuss three distinct approaches to suppressing these pathological fluctuations. We complement this discussion by applying Cheeger's theory relating the geometry of manifolds to their vibrational modes to study the spectra of Hamiltonians. In particular, we present a detailed study of the statistical properties of loop gas and string net models on fluctuating lattices, both analytically and numerically.