Eligibility criteria, information sources, and search strategy

A systematic review of the existing evidence on the effects of double-blind peer review on publication rates by author gender was performed. Initially, an extensive literature search in the PubMed, Embase, Web of Science, and Scopus electronic repositories was conducted. All published literature in the English language from database inception through 2020 was searched, and the search strategy involved the terms “blind,” “peer review,” “gender,” “woman,” and “author” ( Supplemental Table ). The reference lists of relevant articles were also screened to identify further studies that may have been missed in the original search. Clinicaltrials.gov was not searched because the topic being studied was out of scope of that database.

Study selection

Prospective experimental and observational studies comparing double-blind to single-blind peer review strategies that examined the review score or editorial decisions by perceived first author gender were included. Review articles, commentaries, editorials, and letters to the editor were excluded. Studies evaluating publication rates by author gender and editor or reviewer gender but not assessing the practice of double-blinding were also excluded. One additional experimental study evaluating the review of a mock paper with a fabricated author by male and female reviewers was also included as it simulates and explores conditions of potential bias. The search was conducted by 2 independent investigators (A.K.G. and R.J.) to ensure external validity and reproducibility. Both the authors are trained in systematic searching and were supported by or are themselves professional library staff. This systematic review was performed according to a recommended protocol with all methods for data collection and analysis defined a priori . After eliminating duplicates, all references were reviewed on the basis of title and abstract. The full text of each potentially relevant study was then reviewed for inclusion.

Data extraction and synthesis

Data were extracted from full-text articles using a standardized data form. The extracted data were primarily descriptive and included information on study design, sample size, primary outcome, major findings, and scientific discipline. The studies were first characterized broadly into categories on the basis of study design as experimental studies and observational studies. They were then classified according to results as follows: studies that demonstrated a difference in the publication rates of female-authored publications with double-blind review and studies that demonstrated no impact. The method of identifying author gender was also evaluated. No meta-analysis was possible in conjunction with this systematic review, given the wide variety in the types of outcomes reported and the different settings of each study including the scientific discipline and time period.

Assessment of risk of bias

Given the aforementioned study design heterogeneity, no formal risk of bias assessment was performed. The Preferred Reporting Item for Systematic Reviews and Meta-analyses statement was used to describe the results, and the study was registered with the International Prospective Register of Systematic Reviews or PROSPERO.

Study selection and study characteristics

The initial search yielded 1717 titles, and 32 more were identified using citation tracking. Among them, 123 articles assessing gender bias in scientific publishing or double-blinding were reviewed ( Figure ). Of these, 8 studies assessing the practice of double-blind peer review and the impact of perceived author gender were included. Five of these were experimental studies and 3 were observational studies. The full list of included studies by study design is presented in Tables 1 and 2 . In total, these studies evaluated 8253 articles, 5137 of which underwent single-blind review and 3571 underwent double-blind review (some articles underwent both single- and double-blind review on the basis of study design).

Flow diagram of studies identified in systematic review

From : Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71.

For more information, visit: http://www.prisma-statement.org/

Kern-Goldberger. Double-blind peer review and gender bias. Am J Obstet Gynecol 2022 .

The studies utilized a multiplicity of designs. Two experimental studies sent sample manuscripts to peer reviewers and evaluated their recommendations to the editor. One of these—by Lloyd et al—sent a fabricated manuscript about educational research according to the following designations: (1) male reviewer with male-authored manuscript, (2) female reviewer with male-authored manuscript, (3) male reviewer with female-authored manuscript, and (4) female reviewer with female-authored manuscript. The other—by Borsuk et al—sent the same manuscript with 1 of 4 potential authorships—(1) no name, (2) first initial only, (3) male name, and (4) female name—and assessed “reviewer” impressions on an accompanying survey. Another 2 trials sent real articles for review by single or double-blinded reviewers and compared the acceptance rates and review scores. ^, The fifth study by Carlsson et al used real articles submitted to the European Association of Environmental and Resource Economics annual conference and assigned each paper to 2 reviewers who were randomized to blinded or unblinded review. Of studies reporting the breakdown of perceived author gender, 85.3% of articles were authored by males (2463) and only 14.7% by females (426). ^, Carlsson et al do not report manuscript-level data but do note that 487 reviews were completed on articles authored by female first authors, 736 on articles with at least 1 female author, 231 on articles with exclusively female authors, and 1029 on articles with all male authors. Three observational studies of individual journal publications before and after switching from single-blind to double-blind reviewing ^, or directly comparing journals of the same genre that use each type of review were also included in this review.

Synthesis of results

The observational studies by Budden et al and Roberts et al were the only studies to demonstrate increases in the ranking and publishing of articles with women first authors after the introduction of double-blind review. One experimental study found that female-authored manuscripts were more likely to be accepted by female reviewers , and another observational study demonstrated that the acceptance rate for male-authored (but not female-authored) articles was significantly higher with single-blind than double-blind review. In contrast, the remaining 4 articles included in this systematic review—including 3 of the 4 experimental studies—demonstrated no significant impact of double vs single-blind review on editorial decisions by author gender. ^{, , ,}

The included studies used heterogenous outcomes, including acceptance rate by perceived gender and reviewer score or ranking, all of which serve as proxies for actual or potential publication. They cover a variety of scientific disciplines including biology, behavior analysis, computer science, ecology, economics, and linguistics, though notably none cover medical. The methods to identify author gender (when relevant on the basis of study design) included perceptions on the basis of name, internet searches, and lexicons of baby names. Authorship was defined as sole or first-authorship, with only 1 study examining last authorship also.

Principal findings

This is a systematic review of the published evidence assessing the impact of double-blind peer review on the acceptance rates of female first-authorship in scientific journals. There is currently no consensus on best practices for peer review in medical publishing, and the purpose of this study was to identify and review the existing experimental and observational studies assessing the potential role of double-blind review to facilitate gender equity. The findings of this review demonstrate conflicting evidence regarding the impact of double-blind review on peer review decisions by author gender. This may stem from the 30-year time period of these studies, during which social evolution in the scientific community may have altered gender dynamics in publishing or the very disparate disciplines of the included studies. Ultimately, these data corroborate the potential for gender bias in peer review and suggest a benefit of both author and reviewer anonymity in the review process.

Although it is difficult to draw any specific conclusions about the utility of double-blind peer review in reducing gender bias from this array of studies with conflicting results, gender bias is nonetheless a rampant and well-documented problem in medical academia. There is also a logical rationale for considering double-blind review as the gold-standard for unbiased peer review given the inherent inability to provide an unprejudiced review that is only single-blind. In fact, several journals including this have already adopted double-blind review as a policy. Notably, none of the studies of double-blind peer review evaluated here specifically stem from the medical literature. It would be interesting and important to explore the impact of double-blind peer review on author gender in a contemporary medical journal. This would be of particular interest in the field of obstetrics and gynecology, a discipline dedicated to women’s health and where 85% of physicians and an even higher percentage of nurses and advanced practice providers are women.

Comparison with existing literature

Numerous previous reviews have examined longstanding gender inequality in science academia generally and the publishing of scientific literature specifically. An interesting study from 2012 by Moss-Racusin et al asked science faculty members to rank identical lab manager applications; they were randomized to applications with either the first name “John” or “Jennifer.” The “John” applications consistently scored higher than “Jennifer” applications across all areas (competency, hiring potential, and worthy of mentorship) except likability, regardless of the faculty characteristics (age, gender, subspecialty, and seniority), and the “Johns” were offered a salary of $5000 higher on average. Studies such as this demonstrate the prevalence of unconscious gender bias in the otherwise discerning and data-driven scientific world and suggests a possible role for more rigorous blinding in scientific publishing to eliminate such bias. A large study published in 2019 examining more than 23,000 manuscripts submitted to 6 ecology journals from 2010 to 2015 evaluated differential rates in review score, editorial decision-making, and subsequent citations by gender. Although results suggested different scores and publications rates for articles on the basis of author gender, it is difficult to draw meaningful conclusions from studies such as this without controlling for manuscript quality.

Inconclusive evidence exists about the value of double-blind peer review in general. In a systematic review and meta-analysis of 6 randomized controlled trials encompassing 564 manuscripts, Bruce et al found no impact on the quality of the review report or the rejection rate with double-blinding. However, they noted that the confidence interval was wide (0.39– 1.50), with considerable heterogeneity among the different studies. A Cochrane review on the subject of quality improvements to the peer review process similarly found no definitive evidence supporting the benefit of author or reviewer concealment on the disposition of a manuscript (evaluating 9 studies total including both prospective and retrospective studies). Numerous other blinding-related interventions to improve the quality and integrity of the peer review process have also been developed, including triple-blind review, wherein the editor is also blinded to author identity, and open review, the radical opposite, in which all identities are unblinded including the reviewer. Both the processes are intended to increase accountability and fairness in the review process, but there are even less data supporting these practices than for double-blinding.

The challenges inherent in a double-blind peer review system have also been elucidated previously. These include the difficulty of actually maintaining author anonymity, particularly in smaller subspecialty fields where authors and reviewers are more likely to know each other personally, collaborate, or share prepublished work at conferences. The nature of some studies may also render it impossible to truly blind them to the institutional backdrop. Nonetheless, authors, reviewers, and editors have expressed preference for the double-blind review process over other attempts to eliminate bias such as open review. ^,

Future directions for research in this area involve conducting further prospective studies to evaluate the performance of double-blind peer review in obstetrics and gynecology. This can involve well-designed randomized controlled trials of a “mock” paper submitted to groups of reviewers who are single-blinded with fictitious female authorship, single-blinded with fictitious male authorship, and double-blinded to compare reviewer scoring. Journals considering transitioning to double-blind peer review or interested in trialing this practice can also perform pre or post comparisons of female authorship and acceptance rates from groups of reviewers who are single- vs double-blinded. Studies of this nature unfortunately have no ability to control for manuscript quality or other confounding factors such as author or institutional prestige, reviewers’ personal interest in the work, and seniority, among others. Similarly, studies of review strategies that would allow authors or reviewers to opt-in to double-blind review introduce likely insurmountable selection biases. It may also be valuable to explore the role of other methodologies such as explicit and implicit bias training for mitigating bias and their impact on gender equity in peer review and publishing.

Strengths and limitations

The limitations of this study include the substantial inherent differences between the included studies, as discussed earlier, and the inability to formally analyze metaoutcomes. Although some of the studies used experimental or even randomized trial designs representing the highest level of evidence, others were retrospective in nature. The earliest included studies also date back to the 1990s, and the circumstances of potential gender bias at that time may not be generalizable to the present day. Furthermore, none of the studies actually assess the performance of double-blind peer review in a medical journal. Finally, author gender was not identified on the basis of self-report but rather by reviewer, editor, or investigator perception on the basis of author name, which rests on assumptions and biases and may precipitate inaccuracies. The strengths of this study include the fact that this is a comprehensive and systematic review of experimental and observational studies assessing the effect of double-blind peer review on potential gender bias in scientific publishing. It is especially important to disseminate these findings to an obstetrician and gynecologist audience. A previous commentary in a major obstetrics and gynecology journal has already called for the field to reconsider the common practice of single-blind review, and this study furnishes the evidentiary backdrop for further evaluations of double-blind review in the obstetrical and gynecologic literature. Because the existing body of literature examined here derives entirely from nonmedical scientific publications, it is essential that medical journals—including in obstetrics and gynecology specifically—promote further research to examine the gender-related outcomes of any current and potential future peer review practices.

Conclusions and implications

The impact of double-blind compared with single-blind peer review on author gender remains unclear from the published evidence, but half of the studies included in this systematic review reported evidence of gender bias with single compared with double-blind peer review. Therefore, these limited data support double over single-blind peer review to eliminate or at least decrease gender bias. This is particularly important for scientific progress in the field of obstetrics and gynecology, where most of the practitioners and researchers are women. Although reason dictates that double-blind peer review may have significant advantages over single-blind review in terms of reducing gender or other biases, it is unclear how this might work with a contemporary author population in a medical journal. Peer review remains the mainstay of scientific self-regulation and quality assurance, and it is essential that this process itself be subject to rigorous scientific inquiry to assure optimal performance. This study’s findings lay a foundation for future work in this area to identify evidence-based practices for peer review and establish industry standards for best practices to ensure equity.