The proxy record of global temperature shows that the dominant periodicity of the glacial cycle shifts from 40 kyr (obliquity) to 100 kyr (eccentricity) about a million years ago. Using climate model simulations, here we show that the pace of the glacial cycle depends on the pattern of hemispheric sea ice growth. In a cold climate the sea ice grows asymmetrically between two hemispheres under changes to Earth's orbital precession, because sea ice growth potential outside of the Arctic Circle is limited. This difference in hemispheric sea ice growth leads to an asymmetry in absorbed solar energy for the two hemispheres, particularly when eccentricity is high, even if the annual average insolation is similar. In a warmer climate, the hemispheric asymmetry of the sea ice decreases as mean Arctic and Antarctic sea ice decreases, diminishing the precession and eccentricity signals and explaining the dominant obliquity signal (40 kyr) before the mid-Pleistocene transition.