Designing Experiments Toward Shrinkage Estimation

We consider how increasingly available observational data can be used to improve the design of randomized controlled trials (RCTs). We seek to design a prospective RCT, with the intent of using an Empirical Bayes estimator to shrink the causal estimates from our trial toward causal estimates obtained from an observational study. We ask: how might we design the experiment to better complement the observational study in this setting? We propose using an estimator that shrinks each component of the RCT causal estimator toward its observational counterpart by a factor proportional to its variance. First, we show that the risk of this estimator can be computed efficiently via numerical integration. We then propose algorithms for determining the best allocation of units to strata (the best "design"). We consider three options: Neyman allocation; a "naive" design assuming no unmeasured confounding in the observational study; and a "defensive" design accounting for the imperfect parameter estimates we would obtain from the observational study with unmeasured confounding. We also incorporate results from sensitivity analysis to establish guardrails on the designs, so that our experiment could be reasonably analyzed with and without shrinkage. We demonstrate the superiority of these experimental designs with a simulation study involving causal inference on a rare, binary outcome.