Is it time to review, the review?

This is one of a five-part series from Cosmos Weekly, investigating the peer review process.

Academic peer review – that fundamental structure underpinning the very integrity of science itself – is in peril.

The list of retractions and editorial issues of concern, even from the most-respected peer-reviewed journals, swells daily, exposing the underlying problem of expecting peer review to act as the gatekeeper for scientific rectitude and rigour. This is a job for which it is woefully inadequate.

Academic peer review became an integral part of the scientific publishing process in the early 1970s and quickly became synonymous with trustworthiness – both of the journal and of the science itself.

Fast forward about 50 years, and papers are no longer sent through the post, with editorial assistants cutting up and compiling reviewers’ comments to send back to authors.

The sheer number of papers submitted online to journals for review every day has grown to staggering quantities. In 2017, for instance, 10,768 articles were uploaded to the journal Nature – almost 30 per day. Currently, an estimated almost 2 million papers are published annually, in some 30,000 academic journals. That’s a lot of science to get through and a lot of specialists to find for peer review.

Editors of journals cannot possibly be across the nuances of every single specialisation within their discipline, and although journals do have lists of people in the field to which they can direct specific papers, often authors are asked to suggest appropriate reviewers upon submission of their work. As they have for decades, authors are more likely to recommend reviewers who view their work favourably or align to their particular theory more than a direct opponent’s. Although there is obvious potential for conflict of interest (including instances where authors have reviewed their own articles), as the peer-review system currently stands, the dearth of appropriately capable subject-matter experts makes it a necessity.

State of play

Most journals have statements guiding their peer-review processes, which should help navigate these complications and potentially better balance the potential for conflict. However, in many cases, these focus on the overarching criteria of science. Reviewers are typically instructed to review the overall quality and suitability of the research question, approach or experimental design method and results. But they’re rarely told to begin from the assumption that the data or experiment may be entirely fraudulent or fabricated. This is true even of some of the most prestigious journals, such as the Proceedings of the National Academy of Sciences (PNAS), Science and the Journal of the American Medical Association (JAMA), although the journal Nature does allude to the idea, asking reviewers to consider “validity” and identify and report any “flaws” in the manuscript.

As a general rule, reviewing manuscripts is an unpaid job – performed by academics as part of a long-standing tradition that likely arose from necessity rather than pure altruism.

“One of the biggest issues in peer review is the lack of incentive to do a good job,” says medical researcher Dr Hannah Wardill, from the University of Adelaide. “There is no oversight and no training. People are just so thinly spread. None of these factors facilitate a robust and thorough peer-review system.”

The business of academic research is now on an industrial scale. But faced with limited departmental budgets, the pressure to publish, teach, perform outreach and serve on boards all compete with the duty of peer reviewing. Most peer reviewers simply don’t have the time to pore over images or sections of detailed data looking for the hints of fraud.

Along with the journals for whom they are reviewing, most peer reviewers would not consider it in their purview to approach the paper from the viewpoint of possible fraud.

Faced with an explosion in the number of highly specialised science manuscripts plus a decreasing pool of available or suitable reviewers – typically chosen from a pool of authors already published in the field – the traditional system of peer review has been at failure point for some time. Bad science, faked data and fraudulent manuscripts have slipped through the cracks. Clues are missed and things – from fossils to fish and arachnids to Alzheimer’s research – can go badly wrong when they are.

Peer review arose within academia as a “for-us-by-us” way to demonstrate the scientific validity of research methods and findings.

“One of the biggest issues in peer review is the lack of incentive to do a good job.”

Dr Hannah Wardill

Science is undeniably a human endeavour, with peer review held aloft as the gold standard representative of integrity. Peer review itself relies on inputs from people, and the problems that occur from individuals trying to dodge or bypass the process affect real people, too.

As Douglas Adams writes in The Restaurant at the End of the Universe, “to summarise the summary of the summary, people are a problem”.

However, scientists, yet again, are finding ways to hold the field accountable.

Finding solutions to the peer problem

Cosmos received feedback about peer review from more than a dozen senior scientists, along with survey responses from 187 others about the system.

Professor Adrian Barnett, from the Queensland University of Technology, believes there could be a role for automated screening tools that weed out bad science.

Barnett says the number of positive results – that is, results that show a significant trend or relationship – in the academic literature is “ridiculously, unbelievably high”. “It’s simply impossible to show these constant levels of statistically significant difference between what every group is studying,” he says.

He believes journals are reluctant to publish so-called “null result papers”, leading to researchers “either abandoning projects and never publishing or, in some cases, resorting to ‘reanalysing’ their data – dropping particular variables, deleting outliers… there’s lots of techniques you can use to achieve a significant result”.

Barnett is a statistician, and he’s passionate about improving research quality and overall science integrity through the publishing process. Collaborating with an international team of colleagues, he’s been working on algorithms that can detect poor scientific or reporting practices and could potentially be used as a “first pass” for journals as part of the peer review-process.

This is important, explains Barnett, because it’s incredibly time-consuming for individuals to scrutinise the literature – they “can only scour a tiny fraction of what’s out there, and we are experiencing an epic explosion in paper numbers,” he says. “We need algorithms that will automatically check for basic mistakes.”

Aside from identifying plagiarism or otherwise questionable research, the software could also be run as part of the submission process – similar to the way some universities require students to process their work through external software before they submit – and could ultimately help researchers improve their reporting practices and the overall practice of science.

The algorithm might pick up places where an inappropriate graph has been used for the type of data reported, for example, or might identify places where “blinding” is not mentioned, but authors have stated that the trial is “randomised”.

Algorithms can also be set to flag to authors that more information (such as which participants have been left out of trials, not just who was included) could improve how the research is used by medical staff once the paper has been published.

To Barnett, solving the issues of peer review, including the relentless push to publish at any cost, are urgent.

“Science is being looked at right now to solve some of the world’s biggest problems,” he says. “And a lot of us are spending too much of our time not worrying about those problems, just focussing on getting a certain number of papers published, regardless of the quality of those papers.”

Another solution is to ensure there is funding for the process.

Barnett points to the amount spent on quality control in other industries. “Some of them spend up to 20% [of budget] to ensure quality. In health and medical academic research, we spend almost nothing. And journals don’t want to spend the money and neither do universities, because somehow, bizarrely, it’s not in their interest. It’s why we’re in the situation we’re in.”

Pay-per-review?

Better funding could come in the form of paying reviewers – which would be bucking the trend established over the past 50 years. There’s also the raw cost to consider. A recent study published in the Research Integrity and Peer Review journal estimated that reviewers devoted more than 100 million hours to this unpaid and unrecognised work in 2020.

Collectively, that’s about 15,000 years.

The study estimated the monetary value of reviewers’ work in 2020. For US-based reviewers the figure was more than US$1.5 billion, with US$600 million for China-based reviewers and almost US$400 million for those based in the UK. Even if this much money could be found, there’s the threat of conflict of interest if reviewers are paid.

Alternatively, or perhaps additionally, journals might consider employing sleuths like Elisabeth Bik.

Bik is world-renowned for her diligence in hunting down image discrepancies in scientific papers. The work of Bik and other researchers around the world, published on social media and on PubPeer, could potentially be incorporated into and streamline the process of pre-publication peer review. Such an approach would very often require the full release of original data and images.

“Science is being looked at right now to solve some of the world’s biggest problems, and a lot of us are spending too much of our time not worrying about those problems, just focussing on getting a certain number of papers published, regardless of the quality of those papers.”

Professor Adrian Barnett

In a similar vein, pre-publication websites such as the long-standing arXiv website exist to release papers due for publication before they are accepted by a journal. By this system, authors receive feedback from a much larger community of experts than the two or three reviewers who are typically sent a paper before acceptance by a journal. Bik is a huge advocate for pre-publication servers in helping restore and maintain the high standards of academic integrity.

Ivan Oransky, one of the founders of Retraction Watch – a website maintaining a database of journal retractions and editorial expressions of concern – agrees that paid specialists should be part of the solution, noting that this approach would have the added bonus of legitimising their work and ensuring that the process is fair. It would also ensure they are credited.

Who reviews the peer reviewers?

Oransky argues that some sort of oversight is needed – a kind of ‘meta peer review’ of peer review.

Systems of external supervision and policing of research quality exist in some countries but not others, and there is no internationally agreed standard. The USA’s Office of Research Integrity, for example, has been in place for around 30 years, though Oransky admits it’s been a little “clumsy” at times. Following the lead of Denmark, Sweden set up its National Board for Assessment of Research Misconduct in 2019 and was presented with 46 cases in its first year of operation. Australia has no such body.

A key facet to consider is transparency. The existence of an external governing body combined with the work of sleuths like Bik and the open declaration of retractions such as those found on Retraction Watch would bring an openness to peer review and the publication process.

In 2012, the team behind Retraction Watch called for a “Transparency Index”, designed to rate editors and publishers on how willing they are to share their decision-making processes – including peer-review, requirements for raw data, methods for detecting plagiarism and manipulation, processes for dealing with errors or misconduct and clarity and availability of retraction or correction notices. There is now a website called TOP Factor that seeks to do this very thing.

Academics are keen to ensure that lessons learnt are passed onto future generations of researchers.

Some resources already exist, including “What I learned from predatory publishers” by Jeffrey Beall, the original curator of Beall’s List, a compilation of predatory journals and vanity publishers. In it, he details how predatory journals came to be, the efforts to which they go to silence their critics and the ways they cause damage to the institution of science. Websites such as Think, Check, Submit aim to equip researchers with the knowledge and tools to be able to assess the quality and status of a publisher and avoid interacting with untrustworthy journals or publishers.

“My heart breaks for all the scientists who are building their research based on not-real findings.”

Dr Rebecca Jordan

Ultimately, the problems with peer review stem from people. Some individuals want to get around the processes that are designed to ensure checks and balances, and the options available to them are numerous and varied. When these processes are circumvented, it has a real impact on real people – from individuals like Rachael Lonergan, who has believed for 20 years that a cure for Alzheimer’s is just around the corner, to the many dollars in grants and drug development funnelled towards research based on some questionable data, to government policies and the communities governed by them.

Science doesn’t live in a black box

One of the biggest effects is the erosion of people’s trust in scientific processes and research itself. The integrity of science is undermined when people cheat, modify data and vanity publish. Science will continue to undermine itself if it doesn’t get the process of peer review right.

We have seen in recent years the emergence of terms such as “alternative facts” – a clear sign that accepted truth is now open to interpretation. “Do your own research” is a phrase now widely batted around social media, particularly in relation to fringe websites or articles brandished as evidence to support particular, non-mainstream opinions.

“It’s a great step in the right direction,” says Dr Kiara Bruggeman, a biomedical engineer, researcher and lecturer at the Australian National University. “People are linking to a source when posting about the latest cure for COVID – they’re getting the facts right – but too often they’re getting the truth wrong.”

Context is vital for understanding of the facts, explains Bruggeman. “Cherry-picking one Google search result – even if it’s a Nature article – doesn’t replace the value of the understanding and judgement that experts in the field have from exposure to, and experience in, many different aspects of the specialism.”

Scientists themselves become the victims of failures of process as well.

“My heart breaks for all the scientists who are building their research based on not-real findings,” says Dr Rebecca Jordan, a specialist in conservation and genomics at CSIRO.

“As scientists, we’re happy to be proven wrong as new data comes to light and our understanding grows. But we all put faith in what is out there and that what we are doing is taking the next step from a genuine piece of knowledge and understanding.

“To find that’s false would be devastating and so incredibly wasteful of so many dedicated and talented people’s time and efforts.”

“I think the biggest failure regarding peer review is the belief that its role is solely fact-checking and ignoring the other ways in which peers review, learn from, and help to guide each other.”

Dr Kiara Bruggeman

Finally, a reframing of the purposes and process of peer review and how it interacts with the scientific method is desperately needed, says Oranksy, “particularly where it intersects the public’s consciousness”.

“The public are exasperated, they are tired,” he says.

“It seems like everything is see-sawing all the time. It’s getting harder and harder in today’s media ecosystem to really understand where evidence comes from, and, in some ways that could make peer review and the scientific method even more important.

“We’ve done a lousy job of helping people understand the process. Rather than finding definitive facts, science is a way of getting closer and closer to the truth.”

Bruggeman agrees. “Peer review is more than the pre-publication check, and the truth in context is more than the individual published facts,” she says. “‘Correct’ is a direction for science to go, not always the state where science is right now. I think the biggest failure regarding peer review is the belief that its role is solely fact-checking and ignoring the other ways in which peers review, learn from, and help to guide each other.”

Just as you wouldn’t expect every single step on your way home from the pub to be pointing in exactly the right direction, science is a process of gradually building, and of correction – a net drift, on average, towards a specific direction.

“With best intentions, collaborating with other teams and all the confirmation checks possible, we will still get things wrong,” says Bruggeman. “There will be peer-reviewed published papers out there with conclusions that are wrong. Expecting the world of published science and medicine to be entirely factually accurate is ridiculous, but expecting it to be better than yesterday is reasonable.”

---

This story is part of a five-part series from Cosmos Weekly on peer review. Read the other four:

• When peer review fails, people get hurt
• Putting science under the microscope reveals not all is well
• Peer review: where does it lie in the scientific process?
• Scientific fraud, poor research and honest mistakes leads to thousands of retractions

Next week: Clare Kenyon interviews Elisabeth Bik and Ivan Oransky on Monday. Subscribe to our free daily newsletter to be the first to see it.