Ain't Enough Enough? Constraints on the uncertainty and repeatability of sandbox experiments from probabilistic modelling

Random components inhere every experiment and as such prevent its perfect repetition. However, important conclusions on intrinsic variability of a process with possible implications for the likelihoods of events in nature might be drawn from repetitive experiments given a large enough number of observations. In contrast a small number of observations might cause empirical results to be significantly biased due to large statistical uncertainty related to incomplete sampling. But how large is the number of observations necessary to reduce (spurious) statistical uncertainty to (desired) aleatory uncertainty arising from intrinsic variability? We here study the interplay of uncertainties and their dependence on observation parameters using a probabilistic Monte Carlo approach with reminiscence to traditional sandbox experiments. Based on probability distributions of theoretical test populations of measurements of a single parameter of interest (e.g. thrust spacing) we simulate possible experimental outcomes (parameter mean, intrinsic variability) and study their convergence towards the prescribed (physically accurate) value and its precision as a function of experiment number. We found that for typical sandbox experiments (parameter variability CV < 0.3) between five and ten experiments are needed to reduce the test-retest variability (a measure of repeatability) to less than 10% and converge towards the physically accurate value of the parameter. An optimum number of measurements per sandbox experiment is found to be in the order of ten. An increase of this number does not result in significantly better convergence. We compare the simulation predictions with sandbox experiments designed to cross-validate our approach. The approach can be applied in order to optimize the design of experiments in a more quantitative testing environment.