Replication studies are vital, so why are there so few of them?

There is a crisis happening right under our noses in the scientific community that has to do with replicating studies, where researchers work to reproduce the same results of an experiment.

Even though amazing breakthroughs are being made around the world, the actual nitty-gritty of setting up experiments, collecting data, analyzing data and forming a conclusion is, and I think most scientists would agree, unglamorous but vital work.

Scientists and academics, though, face a host of pressures to publish successful and interesting studies for fear of losing funding for future work.

For a teacher, having a larger number of published papers on their resume could spell the difference between achieving tenure or not. For a scientist working in a research lab, a new breakthrough could put the lab on the map and garner more funding or public attention.

This “publish or perish” mindset, a phrase first coined in 1932,  can be so powerful it can motivate researchers to skew their results or tamper with experiments to purposely garner results with more shock value or that ensures success.

A famous example of someone who had tampered with lab research is Dong-Pyou Han, a former Iowa State University scientist that was sentenced to jail time and forced to pay a hefty fine for spiking rabbit blood with human blood while researching potential vaccines for HIV.

Initially, Han admitted that he had made a mistake, but instead of reporting it, continued to spike samples because it was producing false, but positive results.

Scientists will even sabotage each other as a way to get ahead.

One report in the Scientific American discussed one such instance where a researcher was forced to retract his work due to the theft of his lab notebooks and blackmail. Without his notebooks, he was unable to reproduce his work.

Conducting a study, writing an in-depth review of the research, and having that study published is hard enough. Why, then, are replication studies so important and yet so lacking in the scientific community?

The problem is that a large portion of studies aren’t replicable, and replication research is done so little that most of what’s published is simply taken as fact, especially by the media covering new scientific breakthroughs.

One study emphasizing this hugely lacking step in a majority of published research found that two-thirds of social science studies can’t be replicated.

Researchers from the University of Virginia picked out 21 studies from the journals Science and Nature and attempted to reproduce the results from the key experiments conducted in the studies.

Only 13 of the studies were replicable, while the rest resulted in different outcomes.

That makes for a very large number of published results that don’t hold up to further study, so then is the information incorrect or did the journal that published it miss something major?

A lot of it has to do with above-stated pressure to publish and get flashy, engaging, “sexy” science out into the cultural zeitgeist which will, in turn, get recognition and a better chance of getting published again.

It’s also important to realize that scientists are people and people make mistakes. Even the most rigorous and thorough by-the-book studies can yield strange results. It’s not always clear where or if there was a mistake in the data or the experiment.

For example, in 2011, a group of researchers at the OPERA particle detector in Italy’s subterranean Gran Sasso National Laboratory observed a startling phenomenon, a neutrino that moved faster than the speed of light.

The researchers re-checked their work for months and even published a hesitant paper that called for more research.

Only then was it discovered that the fluke was due to a loose fiber optic cable.

A study’s reproducibility, the rarely considered afterthought, is just as important as the conclusion itself. As the OPERA mishap showed, only through countless replications of the experiment was the mistake discovered.

Replicating a study can catch problems that the researchers may not have even realized were there or factor in limitations that the original study authors ignored. Realistically though, calls for further research and replication are rarely met.

There is good news though as it appears that the scientific community is becoming more and more aware of the reproducibility crisis and researchers want changes to be made.

A report was released earlier this year by the Royal Netherlands Academy of Arts and Sciences stating why researchers, universities, and journals need to do more to ensure their results are replicable.

Some recommendations in the report including having academic institutions put more emphasis on training scientists to conduct replication studies. The report suggests that journals require authors to register reports before conducting any experiments or collecting data.

There has also been a push to make raw data compiled from studies readily available which can help further verify a study’s findings and its reproducibility.

Reproducibility needs to be just as central to the process as the other steps in the scientific method, but to achieve this, there must be a major shift in the way researchers, review committees, and the media handle a study’s results.

—

By Kay Vandette, Earth.com Staff Writer