Reproducibility of Statistical Hypothesis Tests

Reproducibility is of great importance for scientific research. While many practical issues affecting reproducibility have received much attention in recent years, the reproducibility of statistical inferences has received rather little attention. This talk presents an overview of our recent development of methods to quantify reproducibility of statistical hypothesis tests using nonparametric predictive inference (NPI). Reproducibility is considered to be a predictive problem, with emphasis on the probability that a repeat of a test will lead to the same conclusion as an actual test. After introducing the basics of NPI, its application to reproducibility of basic tests is illustrated for small sample sizes. For larger sample sizes, or more complicated tests, approximate methods for computation are presented.