Which Type of Statistical Uncertainty Helps Evidence-Based Policymaking? An Insight from a Survey Experiment in Ireland

Which type of statistical uncertainty -- statistical (in)significance with a p-value, or a Bayesian probability -- enables people to see the continuous nature of uncertainty more clearly in a policymaking context? An original survey experiment used a hypothetical scenario, where participants from Ireland were asked whether to introduce a new bus line to reduce traffic jams, given a research report estimating its effectiveness. The treatments were uncertainty information: statistical significance with a p-value of 2%, statistical insignificance with a p-value of 25%, the 95% probability that the estimate is correct, and the 68% probability that the estimate is correct. In the case of lower uncertainty, both significance and Bayesian frameworks resulted in a large proportion of participants adopting the policy (0.82 and 0.91 respectively). In the case of higher uncertainty, the significance framework led a much smaller proportion of participants to adopt the policy (0.39 against 0.83). The findings suggest participants saw the continuous nature of uncertainty more clearly in the Bayesian framework than in the significance framework.