Preprints and the Pandemic: Quality Control for High-Speed Science
COVID-19 research is happening at lightning speed—sometimes at the expense of sound science.
The findings were startling. In an April 30 paper, Stanford researchers estimated that the actual number of COVID-19 cases in Santa Clara County, California, was 50 to 85 times higher than the number of confirmed cases. Eager to share the results as the novel coronavirus rampaged across the U.S., the authors posted the study before peer review on medRxiv, a preprint website, where it attracted media attention but also was widely panned on social media for questionable scientific methods.
The Stanford paper was just one of thousands about the SARS-CoV-2 virus—of varying degrees of quality— that flooded preprint servers this spring, as the COVID-19 epidemic was declared a pandemic. Such platforms make it possible for researchers to post papers as “preprints” before the work undergoes the rigorous peer review process of respected scientific journals.
In an effort to curate and bring a measure of quality control to the avalanche of coronavirus research, faculty at the Bloomberg School formed the Novel Coronavirus Research Compendium to rapidly assess emerging COVID-19 research published in journals and on preprint servers.
Its verdict on the Stanford preprint? The NCRC reviewers agreed that the study did not have sufficient evidence to assert such dramatic underreporting of COVID-19 cases.
Preprint servers have served as a communication medium between scientists in the basic sciences and fields like physics and economics for about 30 years. More recently, medical and public health researchers have joined in, uploading work to new servers launched in the past few years, typically to seek early feedback from other scientists. However, in the midst of the pandemic, the public and the media have also been looking to preprints as sources for the latest news—in some cases reporting on work that is incomplete or even inaccurate.
“The media has been reasonably responsible in saying ‘this is not yet a peer-reviewed study,’ but I’m not sure that the average person really knows what that means,” says Matthew Fox, DSc, MPH, a professor of Epidemiology and Global Health at Boston University School of Public Health. Even if they do, Fox says, they may still make decisions based on media reports.
In the midst of the pandemic, the media have been looking to preprints as sources for the latest news—in some cases reporting on work that is incomplete or even inaccurate.
What we’re seeing now “is an ocean of research of variable reliability,” adds Steven Goodman, MD, PhD ’89, MHS ’87, associate dean of Clinical and Translational Research and professor of Epidemiology and Population Health and Medicine at Stanford School of Medicine.
“A lot of the COVID research is public-facing,” he says. “Preprint servers are now read not just by other scientists but by the public as well. The pandemic has added a sense of urgency. Both scientists and the public are so intensely interested in these findings that we feel we need to get them out as quickly as possible. But speed has a price.”
In response, some journals have sped up their review process, perhaps beyond what is healthy, Goodman says. It’s unclear whether the phenomenon is good or bad, he adds, but in early June, two high-profile papers on COVID-19 treatments were retracted from the Lancet and The New England Journal of Medicine because of questions on the integrity of their data, which Goodman calls “worrisome signs.” Early notice has been moderately important for some papers, helping get the word out quickly on therapeutics like remdesivir, for example, but there is much unreliable information out there as well. “We’re probably in a situation where the balance right now is net positive, but it’s a close call and could change,” he says.
The concept of the NCRC emerged in April when Elizabeth Stuart, PhD, associate dean for Education and a professor in Mental Health, and colleagues became concerned by the rapid pace of coronavirus and COVID-19 research.
“There wasn’t a place to vet the research, signal its strengths and weaknesses, and also curate and summarize it,” Stuart says.
Recognizing the need for trusted high-level summaries of rapidly released COVID-19 studies, Stuart reached out to Kate Grabowski, PhD ’14, ScM ’07, assistant professor of Pathology at the School of Medicine and in Epidemiology at the Bloomberg School, and Emily Gurley, PhD ’12, MPH, an associate scientist in Epidemiology, to head up the effort. Now, more than 50 faculty and students contribute to the project, curating new studies on a weekly basis.
Reviewing teams evaluate studies in eight topic areas including diagnostics, modeling, epidemiology, and vaccines. As of June 7, the site had posted nearly 200 reviews, and garnered 29,449 page views from 7,902 users.
In selecting work for review, the NCRC focuses on empirical research and the generation of new knowledge. The reviewers ask: Will this study be important for a global public health audience? Is this need-to-know information? Does it add anything new to what we already know? They also review papers that attract media attention but may have significant limitations.
The NCRC tackled one study from China, for example, that reported that the drug remdesivir was not effective in later stages of COVID-19, while an NIH press release distributed the same day indicated the opposite was true. The NCRC’s take was that the Chinese study’s conclusions should be viewed cautiously because the study was terminated for insufficient enrollment, the timing of treatment relative to symptom onset was inconsistent, and the tests used to measure viral clearance were not ideal for the task.
Gurley anticipates a continuing need for the NCRC.
“There’s still so much we don’t know about this virus,” she says. “We don’t yet have good examples of places where transmission has been stopped and everything is over. Until we get through that, I think there’s going to be a need.”
As for whether the use of preprints will continue to grow in the post-pandemic era, time will tell, says Goodman.
“It has created a new pipeline in the non-basic sciences that public health and clinical researchers had not used much before,” he says. “I think the inclination will still be to use mainly the journal system once the sense of crisis—and that even immature information has value—passes. … But I don’t think we’ll go back to the baseline we had before.”