Fisher's Mirage: Noise Tightening of Cosmological Constraints in Simulation-Based Inference

We systematically analyze the implications of statistical noise within numerical derivatives on simulation-based Fisher forecasts for large scale structure surveys. Noisy numerical derivatives resulting from a finite number of simulations, $N_{sims}$, act to bias the associated Fisher forecast such that the resulting marginalized constraints can be significantly tighter than the noise-free limit. We show the source of this effect can be traced to the influence of the noise on the marginalization process. Parameters such as the neutrino mass, $\M$, for which higher-order forward differentiation schemes are commonly used, are more prone to noise; the predicted constraints can be akin to those purely from a random instance of statistical noise even using $(1\mathrm{Gpc}/h)^{3}$ simulations with $N_{sims}=500$ realizations. We demonstrate how derivative noise can artificially reduce parameter degeneracies and seemingly null the effects of adding nuisance parameters to the forecast, such as HOD fitting parameters. We mathematically characterize these effects through a full statistical analysis, and demonstrate how confidence intervals for the true noise-free, $N_{sims} \rightarrow \infty$, Fisher constraints can be recovered even when noise comprises a consequential component of the measured signal. The findings and approaches developed here are important for ensuring simulation-based analyses can be accurately used to assess upcoming survey capabilities.