We exploit the nonlinearity arising from the spin-photon interaction in an In As quantum dot to demonstrate phase shifts of scattered light pulses at the single-photon level. Photon phase shifts of close to 90∘ are achieved using a charged quantum dot in a micropillar cavity. We also demonstrate a photon phase switch by using a spin-pumping mechanism through Raman transitions in an in-plane magnetic field. The experimental findings are supported by a theoretical model that explores the dynamics of the system. Our results demonstrate the potential of quantum-dot-induced nonlinearities for quantum information processing.