How quickly does biodiversity rebound after extinctions? Palaeobiologists have examined the temporal, taxonomic and geographic patterns of recovery following individual mass extinctions in detail, but have not analysed recoveries from extinctions throughout the fossil record as a whole. Here, we measure how fast biodiversity rebounds after extinctions in general, rather than after individual mass extinctions, by calculating the cross-correlation between extinction and origination rates across the entire Phanerozoic marine fossil record. Our results show that extinction rates are not significantly correlated with contemporaneous origination rates, but instead are correlated with origination rates roughly 10 million years later. This lagged correlation persists when we remove the `Big Five' major mass extinctions, indicating that recovery times following mass extinctions and background extinctions are similar. Our results suggest that there are intrinsic limits to how quickly global biodiversity can recover after extinction events, regardless of their magnitude. They also imply that today's anthropogenic extinctions will diminish biodiversity for millions of years to come.