The realization of exciton polaritons—hybrid excitations of semiconductor quantum well excitons and cavity photons—has been of great technological and scientific significance. In particular, the short-range collisional interaction between excitons has enabled explorations into a wealth of nonequilibrium and hydrodynamical effects that arise in weakly nonlinear polariton condensates. Yet, the ability to enhance optical nonlinearities would enable quantum photonics applications and open up a new realm of photonic many-body physics in a scalable and engineerable solid-state environment. Here we outline a route to such capabilities in cavity-coupled semiconductors by exploiting the giant interactions between excitons in Rydberg states. We demonstrate that optical nonlinearities in such systems can be vastly enhanced by several orders of magnitude and induce nonlinear processes at the level of single photons.