Reforming scientific publishing to value integrity

August 24, 2016 Ulrike Träger Early Career Research Community

Sometimes working in academia feels like being a gymnast at the Olympics. Not because we’re tumbling through the lab in glittering costumes, but because of the constant pressure to succeed. Gymnasts are tasked to perform more spectacular routines at every competition, while scientists are expected to publish several high impact papers a year. And as seen with French gymnast Samir Ait Said, who broke his leg in a horrific accident at the vault in the 2016 Summer Olympics in Rio, too much pressure can have drastic consequences. While the “publish or perish” culture in science won’t break bones, it does have a negative impact – the prevalence of scientific fraud, leading to an increase in the number of retractions of scientific papers. This is a problem that doesn’t seem to be easing any time soon – the number of papers retracted in 2010 was more than ten times that in 2000.

A study published in PLOS Biology that investigates what factors scientists reputations are judged on gave a clue as to why this problem exists. When comparing a scientist that produces boring, but reproducible studies with a scientist that publishes exciting, but not reproducible studies, the public perceived the boring scientist as “smarter, more ethical, a better scientist, more typical, and more likely to get and keep a job.” Scientists given the same choices agreed that the boring, but certain scientist was smarter, more ethical and the better scientist. But in a departure from the public’s opinion, scientists found the exciting but unreliable scientist to be more likely to get a job and be more celebrated by peers. This is a stark contrast to the public’s view of science, which seems to favor well-done science over flawed science. Worryingly, when scientists were asked which of the two model scientists they would rather be, more said they wanted to be a scientist that produces exciting results, even though the majority knew that publishing reproducible research is better overall. While one survey of 313 researchers does not represent the whole science community, these results paint a surprising picture of scientists’ priorities.

Publish or perish

But it’s hard to lay blame on the scientists – as they are not wrong in their perception that innovative science is more celebrated than replicable science. In our current scientific culture, publishing high impact papers often seem to be all that counts for success. The shortage of tenure-track positions (only about 14%-23% of PhD level biologists, chemists, or physicists hold tenure-track positions after five years) leads to a fierce competition between scientists for those jobs. And the main way scientific success is judged in job or grant applications is the number of high impact publications an investigator has authored.

The pressure to publish one’s work in the top journals is immense, especially for early career scientists who are just starting to make a name for themselves and have few publications. The prospect of a high impact factor publication has even been shown to effect neuronal response and behavior in controlled experimental settings. The results were comparable to reward-based responses normally seen in the context of money, leading the authors to dub impact factors the “currency of science.”

Publishing in high impact journals has disproportional advantages for the authors, compared to publishing the same data in a “lesser” journal. Advantages range from better prospects on the job market and invites to conferences, to better chances on grant applications. This leads some scientists to strive for the perfect, most publishable study. On the way, they may ignore data not fitting their conclusions or leave out negative data. The rush to be the first to publish may also lead to neglect of thorough replication of experiments. The crisis in reproducibility of scientific data is evident, as detailed in a previous post on the PLOS ECR Community Blog. The strong correlation between a journal’s impact factor and the number of papers it retracts, found in a study in 2011, supports the notion that scientists are more likely to take risks and forget scientific vigor in order to publish in high impact journals.

How can we cultivate greater value for sound science?

One common way to assess quality of science is judging the journals in which scientists publish. Impact factors of journals are one popular measure in use. Most scientists agree that impact factors are an imperfect measure of quality, but they are an easy metric. Calculating the average number of citations per article published is easy. By extension, judging careers by a list of publications associated with one number is easy. But when has the easy way ever been the best way? Impact factors are based on citations, but, like all other citation-based metrics in use, it ignores the fact that one in 15 citations is from the authors themselves. Most importantly, however, citation numbers do not account for the type of citations. For example, a paper may be cited 1000+ times but only in publications challenging its findings, meaning the impact factor in this case is not representative of the quality of the research.

Recognizing the shortcomings of impact factors, PLOS instead uses article level metrics (ALMs) to measure the quality of science published in their journals. ALMs measure the quality of a scientific article by not only taking the number of citations in to account, but also press coverage, shares, and views the article receives. By taking into account social as well as academic factors as soon as the article is published, the influence of the article is charted over time prior to the point where the article is cited in other academic literature. While ALMs can share some drawbacks with the more traditional impact factor (e.g. self-promotion on social media or news coverage of poorly done science), they do provide an advantage over impact factors in getting a faster picture of the influence a paper has on both the scientific and general communities.

A second alternative to impact factor is valuing science based on scrutiny from an open access community. A lot of scientific journals only let paying subscribers view their publications, which limits exposure to research published in these journals. Open access policies allow anyone interested in a study to access the research, without barriers. More importantly, not just the access should be free but also the peer review process. Peer review before publication is a key step in checking the quality of science, however the current peer review system is imperfect. I believe that post-publication peer review should be a key process to improve science integrity. Ideally both pre- and post-publication peer review would be made available alongside the published manuscript for increased transparency in the scientific process. A few publications have introduced open reviews including EMBO, BMJ Open and F100research. Alternatively, you can find online journal clubs like PubPeer where articles are discussed post-publishing, or leave comments on articles post-publication.

Currently, a lot of sound science remains unpublished, as negative or inconclusive data are less likely to be published due to reporting bias. A 2010 study in PLOS ONE showed that 82% of papers published between 2000 and 2007 in the United States included positive results only, in spite of the value of negative data. By publishing negative or null results the scientific literature captures a more complete picture of a particular field, and includes more balanced information. I feel a well-done study with negative results deserves the same recognition as a positive one, as it still expands human knowledge and saves resources for other researchers. For example, publishing what isn’t the cause of a given disease will prevent other scientists from spending time and money looking into the same thing. The PLOS Missing Pieces Collection includes negative, null or inconclusive results, and is a great platform for scientists who conduct an experiment and yield a result of this type. In addition, PLOS ONE is a journal that does publish negative, null, or inconclusive results. Replication studies also receive limited recognition in spite of their importance to advancing the scientific field. They are key in validating scientific findings, but few scientists risk doing them as it is hard to publish them for their “lack of innovation,” – a notion we should start to forget.

Boycotting high impact factor journals

Nobel Prize winner Randy Schekman declared his boycott of top notch journals in 2013, arguing that their policies are damaging to science and cause scientists to cut corners. Schekman’s statement started an important discussion, but it is important to note that despite his controversial stance, the repercussions for him are slight. As a well-established principal investigator and Nobel Prize winner, people will read his papers whether or not they are published in a high impact journal, because his career is celebrated in the scientific community. Early career researchers do not share Schekman’s established professional reputation, and may be hesitant to embrace alternatives to the conventional closed-access, impact factor journals when selecting where to publish their findings. Fortunately, there are plenty of communities devoted to reforming the current scientific publishing system and journals committed to transparency. Some of these include: the OpenCon community, which is a group of ECRs advocating for more open science; and in addition to PLOS journals, eLife, Nature Communications and Cell Reports all publish open access science.

While extreme measures such as completely boycotting high impact journals may not be the best solution for all ECRs, there are initiatives devoted to changing the existing science publishing paradigm. Believing that the “challenges facing early-career researchers because of hypercompetition are damaging the efficiency of science,” the Future of Science initiative is a forum lead by post-docs intended for ECRs to discuss the problems they encounter and possible solutions. I encourage all young scientists to take part in the discussion in order to promote change in the scientific community. We need to shift focus back on the science, and not sacrifice ethics and accountability for career advancement.

Featured Image: Image of a gymnast completing a pass on the vault in the London 2012 Olympic Games. Flickr. License: CCBY 2.0

References

Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, Cronin E, et al. (2008) Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias. PLoS ONE 3(8): e3081. doi:10.1371/journal.pone.0003081

Ebersole CR, Axt JR, Nosek BA (2016) Scientists’ Reputations Are Based on Getting It Right, Not Being Right. PLoS Biol 14(5): e1002460. doi:10.1371/journal.pbio.1002460

Fanelli D (2010) Do Pressures to Publish Increase Scientists’ Bias? An Empirical Support from US States Data. PLoS ONE 5(4): e10271. doi:10.1371/journal.pone.0010271

Paulus FM, Rademacher L, Schäfer TAJ, Müller-Pinzler L, Krach S (2015) Journal Impact Factor Shapes Scientists’ Reward Signal in the Prospect of Publication. PLoS ONE 10(11): e0142537. doi:10.1371/journal.pone.0142537

Reich ES (2013) Science publishing: The golden club. Nature 502:291-293 doi:10.1038/502291a

Sauermann H, Roach M (2012) Science PhD Career Preferences: Levels, Changes, and Advisor Encouragement. PLoS ONE 7(5): e36307. doi:10.1371/journal.pone.0036307

Steen RG (2011) Retractions in the scientific literature: is the incidence of research fraud increasing? J Med Ethics 2011;37:249-253 doi:10.1136/jme.2010.040923

Megan Yu (2016) The irreproducibility crisis- An opportunity to make science better PLOS ECR Community Blog

Eva Amsen (2014) What is open peer review? F100 Research Blog

https://pubpeer.com/

http://admin-apps.webofknowledge.com/JCR/help/h_impfact.htm

https://collections.plos.org//missing-pieces

https://www.plos.org/article-level-metrics

https://www.plos.org/open-access

https://www.theguardian.com/commentisfree/2013/dec/09/how-journals-nature-science-cell-damage-science

https://www.theguardian.com/sport/2016/aug/07/fabian-hambuchen-gynastics-injuries-samir-ait-said-andreas-toba

http://www.sciencemag.org/careers/2015/12/people-year-future-researchs-postdoc-activists

http://futureofresearch.org/

http://bmjopen.bmj.com/