Permutation pvalues should never be zero: calculating exact pvalues when permutations are randomly drawn
Abstract
Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which pvalues are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation pvalues published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign pvalues. Although the pvalue understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact pvalues when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice.
 Publication:

arXiv eprints
 Pub Date:
 March 2016
 arXiv:
 arXiv:1603.05766
 Bibcode:
 2016arXiv160305766P
 Keywords:

 Statistics  Applications;
 62G09;
 62G10
 EPrint:
 12 pages, 2 figures