Extreme variations in the direction of Earth's magnetic field contain important information regarding the operation of the geodynamo. Paleomagnetic studies have reported rapid directional changes reaching 1° yr-1, although the observations are controversial and their relation to physical processes in Earth's core unknown. Here we show excellent agreement between amplitudes and latitude ranges of extreme directional changes in a suite of geodynamo simulations and a recent observational field model spanning the past 100 kyrs. Remarkably, maximum rates of directional change reach ~10° yr-1, typically during times of decreasing field strength, almost 100 times faster than current changes. Detailed analysis of the simulations and a simple analogue model indicate that extreme directional changes are associated with movement of reversed flux across the core surface. Our results demonstrate that such rapid variations are compatible with the physics of the dynamo process and suggest that future searches for rapid directional changes should focus on low latitudes.