Living in the Reproducibility Crisis
A fundamental aspect of research is that the work you are doing should be reproducible by other researchers. It is in this manner that scientists can add individual contributions to a collective body of knowledge and assist in solving problems of great importance. When research is published that is not reproducible it can lay an unstable fountain for the knowledge which is built upon it. Inevitably, this foundation gives way and the time and resources of those spent trying to recreate or build upon the irreproducible work is wasted. When the collective body of knowledge that researchers draw from (i.e. the literature) becomes filled with such work we can say that we have entered a “reproducibility crisis”. According to a 2016 survey published in Nature in 2016, 90% of 1,576 respondents from a variety of STEM (including medicine) fields believed there is a reproducibility crisis in scientific literature. The same survey indicated that only 73% of respondents felt that at least half of the papers in their field could be trusted. Confidence in peer-reviewed literature changes between disciplines, with chemistry and physics generally higher than biology and medicine. Yet across all fields 40%, or more, of scientists failed to reproduce key experiments from the literature. Personally, during my training as a graduate student and postdoc I have routinely failed to recreate outcomes of published experiments. These experiences have, without a doubt, have given me a more skeptical eye.
Unfortunately the implications of living in a reproducibility crisis reach beyond failed experiments in the lab. Publications in high impact journals which detail exciting new mechanisms to treat disease are often pursued by companies with efforts aimed at identifying novel therapeutics. However, scientists from Amgen reported in 2012 that they were unable to reproduce the findings of 47 of 53 landmark oncology papers from academic labs. While details were sparse on what “unable to reproduce findings” meant to the Amgen scientists a trio of follow up papers detailing their experimental method on concrete examples (see previous link) provided sufficient evidence for many to view literature through a more skeptic lens. This skepticism runs over into the financial world as well, where an apparent unspoken rule of early-stage venture funds is to consider that at least half of published studies from academic labs will not be able to be replicated by industrial labs. Because of anecdotes like these large companies with the resources to advance proof-of-concept publications to legitimate therapies may be reluctant to partner, spend the capital, and attempt so. Ultimately this hurts prospective patients the most.
As early career researchers it is important we try to understand what is behind this crisis. By doing so we can learn how to best present our own work. Many of us may remember stories from academica of PI’s who are fiercely focused on getting their ideas published in top-tier journals and the vulnerable positions they but their students and post-docs in to do so. Indeed, the 2016 Nature survey results shows that scientists believe legitimate fraud plays a role in nearly half of all cases of irreproducible research. But the real cause is likely far less insidious. Most contention of published data occurs during complex biochemical studies involving living cells, tissues, or animals; areas known to be inherently variable. As an example, under ideal conditions (that is same laboratory, people, tools, and assay) the results of small RNA interfering screen can vary wildly (see previous link) when compared to an experiment attempted five months later. The difference arose from influence of different analysis methodology that had been subtly altered over that time. Here the culprit was poorly written experimental detail pertaining to the handling of data, nothing to do about the science itself. Indeed, a bad habit which has been lamented against in the articles already linked to this article, poor experimental detail requires scientists who wish to recreate literature experiments to make many assumptions regarding experimental design. In these cases when experiments fail it is nearly impossible to identify why. Furthermore, it is a common misconception that if something cannot be replicated then the conclusion must be false. While true for simple, or single, experiments it may not always be the case for conclusions drawn from multiple experiments on complex biological systems.
As early career researchers we should put ourselves in the best possible position to combat reproducibility issues. Already approximately 2/3 of labs have established procedures to account for reproducibility; often as simple and effective as having another lab member confirm the results of key experiments. For example, many chemists fear that reporting low reaction yields may diminish their chances of acceptance to a high-tier journal so they artificially report higher yields. Having another chemist perform the experiment would allow not only as a check on this bad habit, but also would show that the experimental procedure written is clear and can be useful to other scientists. Asking that labmates follow an agreed upon standard operating procedure for handling experimental data can also prevent the reporting of difficult to reproduced results.
It is my personal opinion that this crisis is a result of poor experimental write-ups (perhaps due to urgency of publishing?) and lack of mechanisms for experiments to be reproduced by outside labs (common rebuttal is cost) before publication. As the reader, how would you address this issues? Or do you think there are other reasons for the current crisis?
Featured image is from Pixabay and is used under a Creative Commons License 2.0 (CC BY 2.0).
References
Monya Baker & Dan Penny. (May 26, 2016). Is there a reproducibility crisis? Nature. Vol. 553, 452-454.
Monya Baker. (February 11, 2016). Biotech giant posts negative results. Nature. Vol. 530, 141.
Florian Prinz, Thomas Schlange and Khusru Asadullah. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews.