Open Peer Review Models: Partial, Full, and Collaborative Reviewing
We’re going to jump right into explaining two models of open peer review today that require the identities of authors and reviewers to be transparent, and a spin-off from those models that involves collaboration between the different review stakeholders, which is what Kairos uses in all its stages of review.
In open peer review models that are not crowdsourced (which we discussed in a previous post), the identities of both authors and reviewers are typically revealed to each other during or after the review process. How reviewers and authors engage with each other during the review process differentiates what we would call partial or fully open peer review.
Partially Open Review
In partially open peer review, the identities of authors are not scrubbed from manuscripts in the submission and review process, and reviewers’ identities are provided with their reviews back to authors. This latter half—reviewers revealing their identity to authors—is what differentiates partially open review from single anonymous review, where reviewers’ identities would be kept anonymous.
In the current sphere of peer review, where the double-anonymous model has taken a stranglehold on scholarly publishing within the last 50 years, the partially open peer review model has still existed and been used in some humanities and humanistically oriented social science disciplines for at least 30 years. (We know that because Kairos has always used a form of this model, as we describe in the last section of this post, and we are celebrating the journal’s 30th anniversary this year.)
There’s nothing particularly special or difficult about implementing a baseline version of this review model: Authors save time from having to scrub identifying information from their submissions, and reviewers get to sign their reviews. That’s it. And yet some editors and scholars want to die on the hill of double-anonymous review, saying it preserves the sanctity and rigor of scholarly publishing. To us, that’s some pearl-clutching nonsense, if we may, and we have 30 years of publishing data and expertise to back us up.
The biggest benefit we see to partially open peer review is the lack of gatekeeping ethos reviewers take on. By signing their reviews and taking personal responsibility for their critiques and feedback, reviewers are asked to function as mentors, writing coaches, guides, and consultants instead of Reviewer 2s, gatekeepers, and otherwise critical colleagues hiding behind anonymity.
Anonymity in and of itself doesn’t help the review process in most cases—particularly in the humanities where dialogue and pedagogy are supposedly so highly valued. Having reviewers be able to write directly to authors—which includes the possibility of contextualizing the submission within reviewers’ understandings of an author’s broader publication record—can be of benefit when it’s done purposefully to support publishing efforts, not towards exclusionary practices.
Other disciplines have already determined that more open forms of peer review can lead to higher quality reviews, and also sidestep some ethical challenges made possible by anonymous reviews (like insisting that some cite the reviewer's work, regardless of relevance); see Bruce et al. (2016), Zhang, Smith, and Lobo (2020), or Nature's recent decision that all new publications will be accompanied by referees’ reports and author responses (this is referred to as transparent peer review).
This is where objectors will often say something like “But we gatekeep because we cannot publish but so many pieces a year in the journal!” As editors, we believe that it’s not the reviewers’ decision whether to ultimately publish something, even as reviews help inform our decision. So when editors say the same thing, we might ask how it would hurt their journal to be kinder to authors by asking reviewers to be collegial in their comments, knowing their names will be on it. The process of open peer review lends itself nicely to more inclusive publishing efforts, even if manuscripts are rejected.
Kairos often receives thank you notes on our rejection letters because authors recognize the effort and care we’ve put into reviews. That’s as precious to us in building disciplinary community as the lovely notes from authors we’ve accepted for publication. And we aren’t the only journal using this version of open peer review. We work with many new journal editors who are interested in using this review practice—and we would love to hear if you know of any that are! Some we have worked with who use partially open peer review, with no anonymity and signed review letters, include Ground Works (begun in 2020), and the Journal of Interactive Technology and Pedagogy (begun in 2012). (JITP also uses post-publication review for several of its sections.)
Fully Open Review
Fully open peer review is similar to partially open, in that authors’ and reviewers’ identities are both transparent, but with fully open review, reviewers work directly with authors to provide feedback and work through revision strategies. This is a highly unusual, but not unheard of, option. Journals such as Hybrid Pedagogy were working with fully open peer review as early as 2011. Another model we've seen used for edited collections and some thematic journal issues is to have contributors serve as each others' peer reviewers. In this case, there can be a fair assumption that the contributors are knowledgeable about the main topics in the other chapters or articles. (Sometimes it can be very challenging to find peer reviewers who specialize in niche research areas, particularly interdisciplinary areas that draw on multiple fields; when you gather experts together, there often aren't enough people who aren't contributing who can provide peer review.) Collaboration is an important component of, albeit not exclusive to, fully open peer review. (We give more examples of what this can look like in the final section of this post.)
The biggest benefit to fully open peer review is that it can build strong disciplinary community. Engaging authors in the review process (a time when some authors feel lost and often do not revise and resubmit their work that could have otherwise been a great publication), as well as engaging reviewers in mentoring and bringing up authors into scholarly publishing can only benefit our scholarly communities.
One challenge to fully open peer review is coordinating how reviewers and authors will communicate with each other, although this can be done through ubiquitous platforms and programs (email, collaborative writing spaces, video chat, etc.). There is also the challenge—perhaps the biggest one of all—of training reviewers to be collaborators instead of gatekeepers. In disciplines where pedagogy is a valued component of scholarly life, asking reviewers to function collaboratively and asking authors to be vulnerable in receiving guidance through direct contact may be an easier reach than in disciplines that value rank, classist, and gatekeeping behavior. Peer review criteria and inclusive publishing policies at journals will go a long way to encouraging peer reviews who want to be mentors and see their disciplinary publications thrive.
Kairos Model
At Kairos, we primarily use a partially open model, as we've described it above. Authors are known to reviewers, and we include reviewers’ names on review letters. Authors will already have an idea about who is reviewing their work, as we almost exclusively draw reviewers from our editorial board, which is listed on our masthead.
The process for peer review at Kairos has not changed too much in 30 years. It starts with a review of submissions by our full editorial staff, where we gather on Zoom to evaluate both content and design, and discuss which section it is best aligned with. We take notes about our comments and, once we've reached a consensus, we provide our feedback to the authors, along with a notice about whether (1) the work will be moved to the next stage of the review process (that is, ready to be sent to our peer reviewers), (2) they need to make changes and resubmit, or (3) we have determined that the submission is not developed enough or not a good fit for the journal (essentially a desk rejection). We call this first round of editorial review "Tier 1"— over the past few years we've opened this process up to more members of the staff (it used to be primarily section editors and senior editors) as we find it useful to have more varied viewpoints represented in the conversation.
Once a submission has been approved at the Tier 1 level (although sometimes with suggestions for revisions to complete before sending on to the next stage), we post a request for reviewers to our full editorial board and select a minimum of three reviewers to perform the peer review. We call this Tier 2 in our review process. The reviewers collaborate on their review via email and then the editors synthesize and send the author the signed reviews with any revision requests that need to be made prior to publication. Depending on how many revisions are required, the submission is usually re-reviewed by the same team of peer reviewers before moving into our extensive copy-editing process on the way to publication.
We also have an optional third stage (Tier 3) for authors who receive an R&R at Tier 2, where we provide mentors to work directly with authors to help them realize their vision. This mentor is usually a volunteer from the editorial board who reviewed the submission, but can sometimes be a staff member instead. Authors are given 12 weeks to work with their mentor, and this work is scheduled amongst the author and mentor. Pieces are not guaranteed to be accepted once they are resubmitted, but we do require authors to resubmit to us after the investment of this process.
In the first few years of the journal, Tier 3 was the sole form of review (completely open—authors working directly with reviewers, and where reviewers sometimes could have been considered major contributors to the final product). This format only ran from 1996 to about 1998, and ended because it was far too labor intensive to use for every submission. We switched from that fully open model to one where the full editorial board would collaboratively review each submission, but that also turned out to be somewhat inefficient, and we eventually settled on using a subset of the full board for each peer review.
One of the things we really value about our process is how collaborative it is at each stage—involving all of the editorial staff at Tier 1, the collaboration between reviewers (and sometimes reviewers and editors) at Tier 2, and between authors and reviewers/mentors in Tier 3. These kinds of collaboration tend to reinforce the idea that peer review is meant to lead to improvements in design and argument rather than serving only as a gatekeeping mechanism that purports to evaluate the quality or contribution of the work (although of course, rejection is possible at each stage). Our goal is to guide and improve every submission we publish. For Kairos, collaboration is key in both our partially open (Tier 1 and 2) and fully open (Tier 3) stages of the review process.
Do you know of journals that use partially, fully, and/or collaborative review models? We’d love more examples!
References
Ball, Cheryl E. (2013, January 28). The kairotic nature of online scholarly community building. mediaCommons: a digital scholarly network. https://mediacommons.org/fieldguide/question/how-do-we-build-digital-cohorts-and-academic-communities/response/kairotic-nature-online-sc
Bruce, Rachel, Chauvin, Anthony, Trinquart, Ludovic, Ravaud, Philippe, & Boutron, Isabelle. (2016). Impact of interventions to improve the quality of peer review of biomedical journals: A systematic review and meta-analysis. BMC Medicine 14(85): https://link.springer.com/content/pdf/10.1186/s12916-016-0631-5.pdf
Zhang, Don C., Williamson Smith, Rachel, & Lobo, Sheryl. (2020). Should you sign your reviews? Open peer review and review quality. Industrial and Organizational Psychology 13(1): 45-47.https://www.cambridge.org/core/journals/industrial-and-organizational-psychology/article/should-you-sign-your-reviews-open-peer-review-and-review-quality/C9859BB44B0EB0778D514B3C7F5A48E9 doi:10.1017/iop.2020.5