We speak with Ludo Waltman, Professor of Quantitative Science Studies and Deputy Director of the Centre for Science and Technology Studies (CWTS) at Leiden University, on how the COVID-19 pandemic is changing the way we assess the quality of scientific literature.
What are you working on at the moment?
Ludo Waltman: I am looking at how many COVID-19 papers are being produced and how they are being published. Many are published in peer reviewed journals, but a remarkable number of researchers are making their work public before peer review as preprints. Until now, this has been relatively uncommon in biomedical fields.
Preprints allow research to be disseminated faster compared to the traditional way that takes months. However, because we are not used to using preprint servers for biomedical research, it has raised concerns about the quality of the work. Other researchers, as well as journalists, doctors, policymakers… may not understand the “status” of these preprints and the difference between peer-reviewed and non-peer-reviewed work. This could lead to misinterpretations that might have harmful consequences.
What is happening, in my opinion, is a culture change. Gradually, we are all starting to understand that you need to look at preprints in a slightly different way compared to peer reviewed articles. COVID-19 is accelerating this culture change.
What is happening to COVID-19 preprint articles?
Ludo Waltman: Since January 2020, I have found around 30,000 preprints on COVID-19. Over time, these preprints are turned into peer-reviewed articles that are presumably of higher quality. I say presumably as this is not always the case; some COVID-19 articles have recently been retracted from very prominent journals.
Of course, there are reasons to be careful with preprints, but they are not fundamentally different from journal-published articles. While peer review improves the research quality, it doesn’t always identify all the weaknesses or in some cases even major flaws. It is not a perfect filter to solve all quality problems.
It is surprisingly difficult to find out how many preprints have been turned into journal articles, but it seems that around 20% have been converted. This seems like a small number, but peer review takes time. It is to be expected that preprint articles made available in August-September won’t yet have been turned into peer reviewed articles. When we look at the preprints that appeared at the very early stages of the pandemic in January-February, then we see that about 50% have been converted into journal articles.
We don’t really know what is happening here. At least part of the reason for this is that some of these preprints have major quality problems and it is difficult to find a journal that is willing to publish them. However, I think that there are other possibilities. In some cases, it may be the authors’ deliberate choice to make their work available as soon as possible through a preprint. They may not be interested in going through the process of publishing in a peer reviewed journal, especially if they feel that in a few months their findings may have lost their relevance.
What are publishers doing to improve the peer review process?
Ludo Waltman: There is evidence that COVID articles are being published quicker than non-COVID ones. We are seeing publishers take several initiatives to speed up the peer review process. For example several of publishers have joined the COVID-19 rapid review initiative . Among other things, this initiative organizes transfer of referee reports and articles between journals, also between journals owned by different publishers. I am involved in the evaluation of the initiative and disappointingly, we are learning that the transfer option is hardly being used. This seems to be mainly due to authors not wanting to take up the offer of publishing in a different journal. It is a shame, because it means that when a journal decides not to publish an article that it has sent for peer review, the authors end up submitting it to a different journal and the process starts all over again.
A more promising development involves peer review on online platforms built on top of preprint servers, such as ‘Outbreak Science Rapid PREreview’, that enable researchers to review each other’s work openly. Interestingly, MIT press, a publisher in the traditional sense, has embarked on a similar initiative ‘Rapid Reviews: COVID-19’, establishing a new way of doing peer review.
How can readers determine the quality of COVID research and make best use of the findings?
Ludo Waltman: Most preprint articles include a warning banner stating that the research has not been peer reviewed. The pandemic is forcing everyone, researchers, journalists, policymakers to develop a better understanding of how the publication system works and gradually move away from the dichotomy between research in journals, that has been peer reviewed and is supposed to be reliable, and other types of work that don’t have this ‘stamp of approval’.
Instead, we face a continuum, from really low-quality research that even preprint servers won’t publish, to research in journals that carry out really detailed and in depth peer review at the other extreme. But there is lots in between, research in journals that carry out a more superficial peer review and preprints that have benefitted from lots of comments from experts in the field.
We are still in the process of developing clear markers that inform researchers (and the public at large) about the level of trust you can have on a research output in an efficient way. We need to find a language that is easily understood, by doctors, journalists, policymakers…, and conveys different levels of soundness or trust you can have in the findings. I don’t have the solution, but I think that in a few years we’ll have systems in place for this and that they will be adopted on a large scale.
Further understanding of how research is organised and published will inform better ways of disseminating the information to different sectors of society. I am currently looking at the extent to which the peer review process changes the main conclusions of COVID-19 studies. The results could help determine the value of preprints to policymakers, for example, as they tend to focus on the conclusions rather than the details of a scientific study.
