Enhanced cooperativity for quantum-nondemolition-measurement-induced spin squeezing of atoms coupled to a nanophotonic waveguide

We study the enhancement of cooperativity in the atom-light interface near a nanophotonic waveguide for application to quantum nondemolition (QND) measurement of atomic spins. Here the cooperativity per atom is determined by the ratio between the measurement strength and the decoherence rate. Counterintuitively, we find that by placing the atoms at an azimuthal position where the guided probe mode has the lowest intensity, we increase the cooperativity. This arises because the QND measurement strength depends on the interference between the probe and scattered light guided into an orthogonal polarization mode, while the decoherence rate depends on the local intensity of the probe. Thus, by proper choice of geometry, the ratio of good-to-bad scattering can be strongly enhanced for highly anisotropic modes. We apply this to study spin squeezing resulting from QND measurement of spin projection noise via the Faraday effect in two nanophotonic geometries, a cylindrical nanofiber and a square waveguide. We find that, with about 2500 atoms and using realistic experimental parameters, ∼6.3 and ∼13 dB of squeezing can be achieved on the nanofiber and square waveguide, respectively.