Guidelines: How to Implement Open Peer Review and Foster its Adoption

interview with Dr Tony Ross-Hellauer

In February 2019 the “Guidelines for Open Peer Review Implementation” were published by Tony Ross-Hellauer and Edit Görögh. The publication came up as a result of a workshop held in London about one year ago within the OpenUP EU project. Around 15 publishing industry experts as well as peer review researchers took part. The results of an online survey previously conducted on attitudes towards open peer review amongst academic editors, authors, reviewers and publishers for the OpenAIRE2020 project were also taken into account. The guidelines are made for publishers and for editors. They help to guide the process of implementing open peer review processes and serve as a checklist which covers each step.

In an interview for our blog, Tony Ross-Hellauer gave us some background information and central recommendations. He is the leader of the Open and Reproducible Research Group at Graz University of Technology, Senior Researcher at Know-Center, and Editor-in-Chief of the MDPI open access journal “Publications.”

What are the core elements of open peer review?

In 2017 I published a paper which was a systematic review of definitions of open peer review. I found out that there are a lot of different opinions on what open peer review is. There are many different elements of open peer review, which don’t actually need to be implemented together at all.

The first major element would be opening up reviewer identities. Normally peer review is at least single-blind which means that the authors don’t know who the reviewers are. But in one form of open peer review the authors are aware of the identity of the reviewers. Another form of open peer review is opening up the peer review reports. The third major element would be opening up participation in the review process rather than having reviewers invited by an editor.

And what are the advantages to open peer review?

These major elements have all different advantages and disadvantages. I think they all revolve in some ways around increasing transparency, accountability, or participation of the peer review processes. One of the major concerns with peer review in general is the amount of bias, explicit or implicit, among editors, reviewers, and authors as well. If reviewers weren’t hidden behind this wall of anonymity there might be less chance of biases or conflicts of interest to play out.

One of the major advantages of open reports is that there is a lot of great information within peer reports which at the moment never gets seen. This is great contextual information for scientific articles and scientific works. But it also records the quality assurance process that the scientific work has been through. It shows the rigour of the process.

Finally, allowing open participation in the peer review process could be more inclusive in terms of who is considered a “peer”. Scholarship is often conducted within pretty small and very well defined circles of expertise. But especially with the rise of transdisciplinary and interdisciplinary research there may be people outside those small groups, or event outside the research system who are qualified to usefully comment on a publication, and who might never have been found by an editor.

What are the guidelines most important topics and what are your general recommendations?

The main message is that in open peer review there are a lot of different elements which can be introduced in different ways. With open identities, for example, you can make it optional that reviewers open their identities or you can make it necessary. Or the reviewers remain anonymous until the end of the review process. All these individual elements are separate and have different affordances. The main point is: There are lots of options; there is not a big deal of empirical research which says which one is best in certain circumstances. A journal is like a community and usually you need to respond to the attitudes, the beliefs, and the readiness of the community itself.

So, at a general level you first have to decide what you want to do, what you want to achieve with open peer review. That will help you to select which elements you will actually implement. Having decided on those goals you have to reach out to research communities and get them on board.

Next, plan technologies and the costs involved in publishing peer review reports, for example. There is not an element for that in most manuscript submission systems and there is no easily scalable workflow yet. So you might have to find a workaround or you might actually have to pay. Extra training, explaining the concept to the researchers, and so on also causes additional costs. Therefore some publishers might choose to pilot just some of these processes on a few journals at once. The results of these pilots always seem to be quite positive. Finally, you should evaluate how the pilot went. Consider to make a real study of this experiment – and to open the data to other researchers as well.

What would you recommend to a researcher?

Researchers might like to consult another guideline: Ten Considerations for Open Peer Review. Researchers should consider that when identities are open; it is a different situation for giving and receiving constructive criticism. If researchers engage in open peer review they obviously can get very visible recognition.

Another major thing is that open reports are a great training resource for especially early career researchers. When more of this process is open online, there is less of a mystique to it and early career researchers can find good resources. And finally, if you want to take part in open peer review there is always room to practice it. For example, many journals, although not officially offering “open peer review”, nonetheless allow reviewers to sign their reports if they wish – to be sure, check with the journal editor. You can also take part in commenting on pre-print services

What could libraries do? What is their specific role?

I think libraries, with their information literacy courses and so on, would be a great place for peer review training courses, and also, making researchers and others aware of the benefits as well as of some of the drawbacks of open peer review. But I think honestly that the way forward for libraries is to assume more of the functions of “publishing”. More and more university presses have their own journals, and these systems can be overseen by the electronic publishing unit within libraries and so on. So, I see the role of libraries less and less to bring information into the university, but more and more to assist in disseminating the information or the research that gets created within the universities to the outside.

Where else could open peer review processes be applied?

Obviously they can in principle be applied wherever you apply peer review. There have been experiments with open peer review of books, whether proposals or manuscripts. Open peer review of conferences is increasingly a theme, but also some increased transparency of grant proposals could be useful as well. Though it is really an open question how this could be put into practice.

What else should be done to foster open peer review?

More education is necessary. And also projects like TRANSPOSE, in which I am involved. Transpose is collecting a database of journal policies for open peer review as well as for pre-print policies because there is no central resource that tells you at the moment how many journals actually do open peer review. Our resource should be online soon so that people can actually go to one place and find all the journals that have all these different kinds of open peer review processes in place.

What will be your future work?

There are lots of ideas for future research. I am very interested in how consensus is built within open peer review processes and how levels of power are played out, especially between editors and reviewers. I think that the editor still is really a black box and they have so much executive power to basically accept or reject, but also to steer the process by deciding to choose reviewers who they know will be critical and so on. I am also interested in the differences on a disciplinary level in social sciences, humanities and STEM subjects in regard to open peer review. And why that might mean that we actually might need different forms of peer review for different disciplines. In general I would like to work on initiatives to foster data sharing amongst publishers with research communities so that we can foster a more evidence-based approach to peer review research.