To make science reproducible, focus on prevention not cure

This post was written by Marcus Munafò, University of Bristol, Malcolm Macleod, University of Edinburgh, they are on the steering group of the UK Reproducibility Network and Malcolm Skingle, director of academic liaison at GlaxoSmithKline. 

Improved upstream quality control can make research more effective, say Marcus Munafò and his colleagues.

Science relies on its ability to self-correct. But the speed and extent to which this happens is an empirical question. Can we do better?

The Covid pandemic has highlighted existing fault lines. We have seen the best of scientific research in the incredible speed at which vaccines have been developed and trialled. But we have also seen a deluge of Covid-related studies conducted in haste, often reflected in their less-than-ideal quality.

Peer review, the traditional way of assessing academic research, occurs only after work has been done. Can we identify indicators of research quality earlier on in the process when there is more opportunity to fix things? And if we did, could scientific knowledge be translated into societal benefit more rapidly and efficiently?

In many ways, the cultures and working practices of academia are still rooted in a 19th-century model of the independent scientist. Many research groups are effectively small, artisanal businesses using unique skills and processes.

This approach can yield exquisitely crafted output. But it also risks poor reproducibility and replicability—through, for instance, closed workflows, closed data and the use of proprietary file formats. Incentive structures based around assessing and rewarding individuals reinforce this, despite the welcome shift to team-based research activity, management, dissemination and evaluation.

Lessons from industry 

Research needs a more coherent approach to ensuring quality. One of us has previously argued that one way to achieve this would be to take the concept of quality control used in manufacturing and apply it to scientific research.

Pharmaceuticals are one R&D-intensive industry that has worked hard to improve quality control and ensure data integrity. Regulatory frameworks and quality-assurance processes are designed to make the results generated in the early stages of drug development more robust.

Indeed, some of the early concerns about the robustness of much academic research—described by some as the ‘reproducibility crisis’—emerged from pharmaceutical companies.

For regulated work, major pharmaceutical companies must be able to demonstrate the provenance of their data in fine detail. Standard operating procedures for routine work, and extended description of less common methods and experiments, makes comparisons between labs easier and improves traceability.

Data constitute the central element of robust research. The integrity of the systems through which data are collected, curated, analysed and presented is at the heart of research quality. National measurement institutes, including the UK’s National Physical Laboratory and National Institute for Biological Standards and Control have a role to play, sharing best practice and developing protocols that contribute to international standards.

How well these systems perform depends on many factors: training in data collection and management; transparency to allow scrutiny and error detection; documentation, so that work can be replicated; and standard operating procedures to ensure a consistent approach.

Red-tape review

Academic researchers are increasingly keen to learn from industry, and vice versa—to identify best practice and ways to implement higher standards of data integrity. University and industrial research are very different, but academia can learn lessons and adopt working practices that might serve to improve the quality of academic research in the biomedical and life sciences.

Learning from other sectors and organisations is a central theme of the UK Reproducibility Network. The network, established in 2019 as a peer-led consortium, aims to develop training and shape incentives through linked grassroots and institutional activity, and coordinated efforts across universities, funders, publishers and other organisations. This multilevel approach reduces the cost of development and increases interoperability, for example, as researchers move across groups and institutions.

Given the likely future pressures on the UK’s R&D budget, effective and efficient ways to bolster research quality will be essential to maximising the societal return on investment. Simply encouraging, or even mandating, new ways of working is not sufficient—many funders and journals have data-sharing policies, for example, but adherence is uneven and often unenforced.

A coordinated approach will require a clear model of research quality; buy-in from institutions, funders and journals; infrastructure; training; the right incentives; and ongoing evaluation. Coordinating all these elements will be challenging, but it is essential to improving research quality and efficiency. We need to take a whole-system approach.

This also applies to the independent review into research bureaucracy recently announced by the UK government, charged with identifying how to liberate researchers from admin. This is laudable—academia should certainly not be regulated in the same way as the pharmaceutical industry—but the review should recognise that an ounce of prevention can save a pound of cure.

Developing and deploying systems that improve research quality might increase efficiency and reduce research waste, as well as securing greater value for our national research effort.

This blog post was originally posted on Research Fortnight, you can read the original article here.

Experts aren’t just for emergencies: How COVID-19 is changing evidence-based policy making for the better



This blog post was written by Anthea Terry, Interim Head of PolicyBristol and was originally published Universities Policy Engagement Network (UPEN) blog. Read the original article.

Michael Gove famously said in 2016 that ‘people in this country have had enough of experts’, and with social media ‘bubbles’, fake news, and the media desire to present opposing viewpoints – however marginal – it can often feel this way.

But the actual public perception of experts and their work is more nuanced. A 2018 survey by the Wellcome Trust found that 82% of people said they were fairly or very interested in health research, up from 77% in 2015, and 75% in 2012.

The value placed on experts by policymakers has always been variable and hard to measure, ‘evidence-based policymaking’ has been around for decades, and for almost as long, the perhaps inevitable cynicism about ‘policy-based evidence making’. We have incredible success stories about research influencing policy (my favourite being the research on CFCs that led directly to the Montreal protocol and recovery of the ozone layer), yet the combined weight of almost all the world’s climate scientists fails to enact sufficient policy change.

One of the many unique features of this time is the level of public discussion about research and the role of experts in policy making. I can’t remember another time when the membership of expert advisory groups such as SAGE was mainstream news. Similarly, a call for participants in a Covid-19 vaccine trial in Bristol was shared on neighbourhood WhatsApp and Facebook groups, and I’m talking to my family about R-numbers and logarithmic growth curves whilst lamenting the lack of supermarket delivery slots.

Continue reading

A Day in Parliament: Kate Oliver STEM Finalist 2018

Image Credit: STEM4Brit

A day in Parliament: Kate Oliver, STEM Finalist 2018

On the 12th of March I went to Parliament, for the second time in my life, this time accompanied by a rolled up piece of A1 paper. I was going to ‘the major event bringing early career researchers and parliamentarians together’, STEM for Britain*.

This poster session, now in its 21st year following its founding by Eric Wharton MP, invites around 50 exhibitors in each of Physics, Chemistry, Mathematics, Engineering and Biological sciences to explain their work to the employees of Parliament and a panel of expert judges. Five of us from Bristol had been selected to present – around a third of applications are successful – all in different categories, and we had been preparing our two-minute pitches for a few weeks, with the help of our supervisors, university support staff, and patient friends. Continue reading

Brexit: can research light the way?


Cressida Auckland, a Parliamentary Office of Science and Technology (POST) Fellow

Cressida Auckland, a Parliamentary Office of Science and Technology (POST) Fellow

Chandy Nath, acting Director of the Parliamentary Office of Science and Technology (POST)

Chandy Nath, acting Director of the Parliamentary Office of Science and Technology (POST)

What could Brexit mean for UK science? What impact will it have on UK fisheries? Could Brexit be bad news for emissions reductions? These were just some questions discussed at a Parliamentary conference last week, organised by the Parliamentary Office of Science and Technology (POST), the Commons Library and Parliament’s Universities Outreach team.

MPs researchers, Parliamentary staff and academic researchers from across the country came together to consider some of the key policy areas affected by the UK’s decision to leave the EU.

Continue reading

Scientific research and the European Union – how UK science may be affected if we choose to leave

Dr ewan fowler

Dr Ewan Fowler, Research Associate in the School of Physiology, Pharmacology and Neuroscience, University of Bristol

The referendum on Britain’s membership of the EU is fast approaching and as the debate intensifies, science is a topic that remains very much overlooked, despite its importance to the UK economy.

I have recently begun to consider the scientific relationship that Britain has with the EU and how UK science may be affected if we choose to leave.  This relationship is not trivial, according to OECD figures the EU produces around 1.7 million scientists, which is more than either China (1.5 million) or the US (1.3 million).

To facilitate this each member state contributes towards a fund called Horizon 2020, which the European Research Council (ERC) distributes to research and infrastructure projects.  The expected budget of Horizon 2020 from 2014-2020 is over €80bn, an increase from the previous incarnation called Framework Programme 7 which had a budget of €53bn from 2007-2013. For projects involving international collaborations a single application to the ERC is required removing the need for separate applications to national funding agencies.

The UK received €8.8bn under Framework Programme 7 from 2007-13, amounting to 3% of total research spending.  This may seem small however it is just shy of charity-funded research (5%) and is typically viewed as a main source of funding for biomedical research.  The UK is highly competitive in obtaining funding as it is currently awarded the greatest number of grants under Horizon 2020, and achieved the second greatest number under Framework Programme 7.

Credit - JISC, Creative Commons

Credit – JISC, Creative Commons

Continue reading

Why do people reject science? Here’s why …

Professor Stephan Lewandowsky, Chair in Cognitive Psychology

Professor Stephan Lewandowsky, Chair in Cognitive Psychology

The State of Science: Why do people distrust science? Why do some of us reject consensus on a whole range of scientific findings? As Professor Stephan Lewandowsky explains, it often comes down to the way we look at the world.

Before the warning labels became mandatory in the US, some 500 cases were reported annually; today, less than a handful of cases are reported each year.

What does Albert Einstein’s theory of general relativity have to do with the human papillomavirus vaccine (HPV)?

What does acid rain have to do with the fact tobacco smoking causes lung cancer?

What does Reye’s syndrome have in common with the CFCs that caused the hole in the ozone layer?

And what do all those issues have to do with the fact our climate is rapidly changing due to human greenhouse gas emissions?

Continue reading