On mitigating the analytical limitations of finely stratified experiments
Abstract
While attractive from a theoretical perspective, finely stratified experiments such as paired designs suffer from certain analytical limitations not present in blockrandomized experiments with multiple treated and control individuals in each block. In short, when using an appropriately weighted differenceinmeans to estimated the sample average treatment effect, the traditional variance estimator in a paired experiment is conservative unless the pairwise average treatment effects are constant across pairs; however, in more coarsely stratified experiments, the corresponding variance estimator is unbiased if treatment effects are constant within blocks, even if they vary across blocks. Using insights from classical least squares theory, we present an improved variance estimator appropriate in finely stratified experiments. The variance estimator is still conservative in expectation for the true variance of the differenceinmeans estimator, but is asymptotically no larger than the classical variance estimator under mild conditions. The improvements stem from the exploitation of effect modification, and thus the magnitude of the improvement depends upon on the extent to which effect heterogeneity can be explained by observed covariates. Aided by these estimators, a new test for the null hypothesis of a constant treatment effect is proposed. These findings extend to some, but not all, superpopulation models, depending on whether or not the covariates are viewed as fixed across samples in the superpopulation formulation under consideration.
 Publication:

arXiv eprints
 Pub Date:
 June 2017
 arXiv:
 arXiv:1706.06469
 Bibcode:
 2017arXiv170606469F
 Keywords:

 Statistics  Methodology