Ask the seminar a question! BETA
How Current Research Practices Are Damaging Science And How Open Science Offers Some Solutions
Prof. Dorothy V M Bishop FBA, FMedSci, FRS
Dorothy V M Bishop
University of Oxford

Quasi-cumulative science - where we think we are building on foundations of prior work, only to find the earlier work isn't solid - has a detrimental effect on science. There is no single cause for this, but rather an unfortunate combination of factors. Five of these, together with possible solutions, are discussed:

  1. Publication bias - One solution is Registered Reports, which decouples decisions about publication from knowledge of research results.

  2. Citation bias. Creates the impression of widespread agreement on a topic, because literature that does not support it gets forgotten - Systematic reviews are one method that attempts to redress the imbalance in citations, but it is not infallible. Unlikely to change unless we increase awareness of the serious consequences of citation bias, and train researchers to seek out and evaluate evidence contrary to their position.

  3. P-hacking, or selective reporting of only positive findings from within a study, often after exploration of many different analysis options. I discuss one form of p-hacking which might be termed 'moving goalposts', where studies that appear to replicate an initial finding actually fail to do so, but instead present results on a related question - Registered reports offer a solution to p-hacking.

  4. Low statistical power, often due to small sample sizes - Training scientists to explore simulated data is one way to counteract underpowered studies.

  5. Obsession of funders/journals with novelty. A 'top down' influence that produces a distorted incentive structure for scientists, who avoid slow, careful science that builds up knowledge on a topic, and rather feel they must overhype results and jump from one hot topic to another - The solution is for funders and institutions to change how they evaluate researchers, to focus on rewarding those who adopt open, reproducible methods. The Hong Kong Principles for assessing researchers are introduced as one such approach.

References
  • 1.
    D. Bishop (2019) Rein in the four horsemen of irreproducibility. Nature
  • 2.
    D. Bishop (2020) How scientists can stop fooling themselves over statistics. Nature
  • 3.
    R. Border et al. (2019) No Support for Historical Candidate Gene or Candidate Gene-by-Interaction Hypotheses for Major Depression Across Multiple Large Samples. American Journal of Psychiatry
  • 4.
    C. D. Chambers et al. (2015) Registered Reports: Realigning incentives in scientific publishing. Cortex
  • 5.
    Y. A. d. Vries et al. (2018) The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological Medicine
  • 6.
    A. D. d. Groot (2014) The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica
  • 7.
    S. A. Greenberg (2009) How citation distortions create unfounded authority: analysis of a citation network. BMJ
  • 8.
    A. G. Greenwald (1975) Consequences of prejudice against the null hypothesis.. Psychological Bulletin
  • 9.
    J. J. Locascio (2017) Results Blind Science Publishing. Basic and Applied Social Psychology
  • 10.
    D. Moher et al. (2020) The Hong Kong Principles for assessing researchers: Fostering research integrity. PLoS Biology
  • 11.
    S. B. Nissen et al. (2016) Publication bias and the canonization of false facts. eLife
  • 12.
    R. Rosenthal (1979) The file drawer problem and tolerance for null results.. Psychological Bulletin
UKRN logo
UK Reproducibility Network Webinars
UKRN (UK Reproducibility Network)
Cite as
D. V M Bishop (2021, May 3), How Current Research Practices Are Damaging Science And How Open Science Offers Some Solutions
Share
Details
Listed seminar This seminar is open to all
Recorded Available to all
Video length 54:56