Improving Reproducibility in the Lab

Science is based on evidence, but the scientific research field is facing a reproducibility crisis.

The science journal Nature published a survey in 2016, which demonstrated more than 70% of researchers could not replicate their peers’ studies in well-controlled and standardized conditions. That’s a problem.

While the pharmaceutical and biopharmaceutical areas have made incredible advances in both technology and science, lack of reproducibility of published studies remains a concern. Why is it important? Why does the problem persist, and how can we improve our approach?

The importance of reproducibility

Studies into low reproducibility investigated 53 projects surrounding cancer and found that the primary findings could only be reproduced 11% of the time. Funding and resources allocated to these projects are wasted, along with scientists’ time. In the US alone, research has shown that $28 million is spent every year on preclinical research that can’t be reproduced.

Apart from the cost, this is a worrying statistic, considering we depend on reproducibility in the lab to trust in research. Not only does this foster a lack of confidence in the scientific method, but faulty studies also give patients waiting for potentially life-saving treatments false hope.

In terms of time wasted, any study that has potential clinical applications will be replicated before going on to preclinical trials. For a single study, this can take anything from 3 months, to 2 years and cost upwards of $500,000.

Improving reproducibility in drug development and research would remove risks, make investments count, and increase productivity of research, speed and efficiency. All these factors combined would result in a significant boost in ROI.

Challenges to reproducing studies

What are some of the potential reasons for this lack of reproducibility in the lab?

• Scientists must account for every aspect of an experiment. Living organisms are complex on their own and come with variables – gender, age, strain, housing conditions, breeding or upbringing, and diet are just a few of a long list of possible variables used in experiments.

And then there’s the way the individual scientist will run the experiment and the instructions they follow, whether it is different protocols or SOPs. Even when the parameters and study design are near identical, a single modification can lead to different results.

• To avoid false calculations, scientists must choose the correct study design and pick an appropriate statistical analysis for their sample size.
• Biological reagents also contribute to the lack of reproducibility. Often, the quality control of such reagents is not enough, or they are mislabeled. For example, cell lines used in biomedical and biotherapeutic research are often misidentified and even contaminated.
• Outliers are not given the attention they deserve. It is a well-known phenomenon that scientists are inclined to see the results that fit neatly into their hypothesis as more viable compared to those that don’t support their theory. This bias means that negative results of an experiment are rarely published.
• Increasing competition and pressure in the field to publish full and conclusive data means results that contradict theories are often disregarded.

Recommendations on improving reproducibility rates

How can we boost the reproducibility of a study? The National Academies of Science, Engineering, and Medicine recently published a few recommendations on the subject:

First, focusing on a robust study design by improving training programs and strengthening the understanding of best practices can really make a positive difference. Also, institutions should ensure that students and researchers are properly trained in statistical analysis to avoid common errors.

Second, it is encouraged to adopt vendors that provide validated biological reagents and reference materials. Along with reducing the chances of misidentifying a reagent, there is also a smaller risk of contamination, thereby ensuring quality control.

Third, collaboration is key. To foster a culture of best practice, the pharma/biopharma industry, government, journals and academia must work together and agree upon a standardized set of rules and guidelines.

Fourth, each project should have its findings confirmed before it moves on to further studies, especially if it’s costly and will take a long time to complete.

Fifth, there should be full transparency and traceability on the materials and methods used, as well as the protocols followed in the experiment. This point is logical, as scientists need this comprehensive information to replicate the experiment, in addition to statistical analysis, study design (animal used, sex, age, strain, diet, if the study was blinded and/or randomized) and, where possible, the raw data.

Sixth, it’s important to include all results – even if they do not support the working hypothesis. This can prevent false conclusions and misinterpretations of the data, and reveal opportunities for further research.

How software can help

Improving reproducibility is a challenge that can be approached from multiple angles, including using technology to solve the issue. The E-WorkBook Cloud is an integrated data management platform that addresses these concerns head-on.

When data is tucked away in disparate Excel files and paper lab notebooks, there is always the risk of leaving information out of the report. Traceability is vital to reproduce a study – tracing the samples, materials and equipment back to their origin and throughout their journey builds a complete story.

How data is recorded also has a significant impact on quality. Data should be meticulously documented, including all data points, whether they fit into the hypothesis or not. To replicate work from start to finish, scientists need to be able to access all the experimental information and statistical analysis.

E-WorkBook offers a single platform to store all the relevant information associated with the experiment, along with a powerful search tool to sift through it all and filter by specific criteria in seconds. With all the data in a single location, scientists can trace their samples back to see their genealogy and the built-in SOPs ensure adherence to protocols.

With the E-WorkBook Cloud, researchers have all the tools and information at hand to reproduce an experiment, saving both money and precious time so that patients can get lifesaving treatments sooner.

Conclusion

Lack of reproducibility has led to delays in lifesaving therapeutics, higher treatment costs and tighter budgets. There are numerous studies on the lack of reproducibility, and from them, one common theme emerges: reproducibility is too important to ignore. It might be challenging to improve, but the cost to life sciences as well as patients has proved its worth.

Software, such as the E-WorkBook Cloud, can make reproducibility easier, improving confidence and trust in science and providing a window of opportunity for further studies.

To discover more about the E-WorkBook Cloud and how it can help you improve reproducibility in your lab, contact our team of experts.