Objective
We sought to determine the extent to which reviewers’ recommendations influence the final editorial disposition of manuscripts submitted for publication.
Study Design
Five reviewers retrieved their electronic databases of obstetrical manuscripts that they had reviewed for Obstetrics and Gynecology and the American Journal of Obstetrics and Gynecology. The recommendations of each reviewer were grouped in 1 of 3 categories: rejection (or not acceptance), acceptance with major revisions, and acceptance with minor or no revisions. These recommendations were contrasted in the final editorial disposition of the manuscript, which was recorded as “accepted” or “rejected.” The quality of the reviews was assessed in a random sample of 10% of the reviews, stratified by reviewer and journal.
Results
A total of 635 reviews were analyzed. Overall, the most influential reviewers’ recommendation was rejection, which was accompanied by 93% rejection rate. Recommendation for acceptance with minor or no revisions was accompanied by 67% acceptance rate whereas acceptance with major revisions was accompanied by 40% acceptance rate. There were no variations among reviewers regarding their degree of influence with respect to the final disposition of the manuscript. The final disposition of manuscripts was not influenced by the quality of the reviews nor reviewer’s demographics including reviewer’s age, year of first peer review, and years active in peer review.
Conclusion
The degree of influence on the final disposition of the manuscript depends on the type of recommendation. A recommendation for rejection was the most influential and it was associated with a high rate of rejection. Recommendations for acceptance or minor revisions were also influential but to a lesser degree.
Obstetrical practice and the understanding of the biology of pregnancy and mechanisms of disease are based on peer-reviewed publications. The 2 most influential journals in obstetrics have been Obstetrics and Gynecology and the American Journal of Obstetrics and Gynecology , which have published most of the “citation classics,” as reported by Brandt et al and by Romero et al.
The selection of manuscripts for publication in these journals is based on a combination of the advice of reviewers and the editors’ decision. However, there is a paucity of information about the influence of a particular recommendation by a reviewer to decline, accept, or revise a manuscript on the final editorial decision. Generally, reviewers are asked to choose one of the following options: accept without revision, accept after minor revisions, accept after major revisions, and reject for publication.
Since the early 2000s, it has been customary for the editors of many journals to notify electronically their reviewers regarding the final disposition of each article that they review. This has resulted in a database for each reviewer, so that a possible correlation between the recommendation suggested by the reviewer and the final disposition of the article can be established. This can determine the degree of influence of the reviewer’s recommendation, if any, on the final disposition of the manuscript. Since there is >1 reviewer for each submitted article, it is expected that the degree of influence or the correlation with the final disposition of the article may vary from reviewer to reviewer or even from journal to journal. This information may be of value to reviewers when making a recommendation to the editors when conducting future peer reviews.
The aim of this retrospective study was to determine the relationship between the recommendation of reviewers on the editorial decision to accept or decline an obstetrical manuscript for publication.
Materials and Methods
Five of the authors (A.M.V., C.V.A., S.P.C., J.C.S., and Y.O.) reviewed the electronic files for obstetrical manuscripts they had reviewed from February 2002 through May 2013 for 2 journals, Obstetrics and Gynecology and the American Journal of Obstetrics and Gynecology . The recommendations of each reviewer for every manuscript were grouped in the following 3 categories: rejection (or not acceptance), acceptance with major revisions, and acceptance with minor revisions or acceptance with no revisions (there were very few recommendations of acceptance with no revision). The outcome was the final editorial disposition of the manuscript, which was recorded as accept or reject.
The reviewers’ characteristics considered as possible confounders included reviewer’s age, number of years serving as peer reviewer, postgraduate training in epidemiology or biostatistics, and quality of reviews. To determine the quality of the reviews that the reviewers had provided, a random sample of 10% of the reviews of each of the 5 reviewers was chosen for quality analysis. This random sample was stratified by the journal type ( Obstetrics and Gynecology and the American Journal of Obstetrics and Gynecology ), and restricted to regular submissions or to manuscripts for which data were presented at annual meetings (review articles were excluded from quality review to reduce confounding due to different article types). This stratification ensured that both journals had approximately 10% of the random sample for every reviewer. The quality of the so-chosen reviews was assessed by A.O.O. by using the review quality instrument proposed by van Rooyen et al. The review quality instrument contains 8 questions and had to do whether or not the reviewer commented on: importance of the research question, originality, strengths and weaknesses, organization of the manuscript, interpretation of the results, whether the reviewer used constructive comments, whether the reviewer used evidence to substantiate comments, and overall quality assessment. The strength of each determination was expressed on a scale 1-5, with 5 being the highest quality indicator; thus, the possible range of the total score for each review ranged between 8–40. A.O.O. was blinded to the reviewer, the journal for which the review was provided, and the final editorial disposition of the corresponding manuscripts.
We examined if there were differences in the quality of review scores across reviewers by using the exact Savage scores test and if there were differences across reviewers in the proportion of manuscripts that were recommended for rejection, major revisions and minor revisions or acceptance by using the Fisher exact probability test.
To determine the degree of influence of the reviewers’ recommendations on the final editorial disposition of the manuscript, the data were analyzed separately by each reviewer and journal. Each reviewer’s recommendations were used as predictors of the final disposition of the article. The influence of the reviewer’s recommendation on the final disposition of the manuscript was based on the percent agreement and the degree of concordance. The percent agreement was derived by the number of instances a reviewer’s recommendation were similar to that of a final editorial disposition. Concordance, on the other hand, was expressed as a kappa statistic with associated 95% confidence interval. These data exhibited 2 issues. First, these data were unbalanced since the final editorial disposition of a manuscript for the purpose of this analysis was rated on 2 levels (accept or reject a manuscript) while the reviewer’s recommendation was based on 3 levels (reject, major revisions, or minor revision/accept). To overcome these unbalanced data, we estimated the kappa coefficient based on the method proposed by Crewson. Second, for small samples or when the kappa statistic approaches the null (ie, no concordance), the variance estimate of the kappa coefficient can become unreliable. To overcome this bias, we estimated the kappa coefficient and its bias-corrected 95% confidence interval based on bootstrap resampling methods with 5000 replications without replacement. This bootstrap resampling was implemented in software (SAS, version 9.4; SAS Institute Inc, Cary, NC). Lastly, we fit a logistic regression model to evaluate if the final editorial decision was influenced by the reviewer’s recommendation before and after adjusting for potential confounders. These confounders included the quality of review score, year of first review, and the number of years active for each of the 5 reviewers.
Results
The total number of manuscripts reviewed, reviewers’ recommendations, year of first review, number of years active, and quality of review scores for each of the 5 reviewers are listed in Table 1 . Among the 5 reviewers, the recommended rejection rates ranged between 38–54%, the recommended acceptance with major revisions ranged from 17–38%, and the recommended acceptance without or minor revisions ranged from 16–34%. There were significant variations in the type of the reviewers’ recommendations ( P < .001). The quality of review scores also showed significant differences across the 5 reviewers, both overall ( P = .002) and for 7 of the 8 questions.
| Characteristic | Reviewers a | P value | ||||
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| Total manuscripts reviewed | 220 | 121 | 110 | 100 | 84 | – |
| Reviewer’s recommendation, % | < .001 | |||||
| Rejection | 54 | 38 | 50 | 49 | 43 | |
| Acceptance with major revisions | 30 | 38 | 30 | 17 | 30 | |
| Acceptance or minor revisions | 16 | 24 | 20 | 34 | 27 | |
| Year of first review | 1983 | 1992 | 2003 | 1995 | 1994 | – |
| No. of years active | 30 | 18 | 24 | 19 | 24 | – |
| No. of manuscripts for quality assessment b | 20 | 10 | 6 | 10 | 6 | – |
| Quality of review score c | ||||||
| Importance of research question | 1 (1–4) | 4 (3–5) | 2 (1–4) | 3 (2–4) | 4 (1–5) | < .001 |
| Originality | 1 (1–4) | 1 (1–3) | 1 (1–5) | 2 (1–4) | 1 (1–3) | .678 |
| Strengths and weaknesses | 3 (1–4) | 5 (3–5) | 3 (2–4) | 3 (2–4) | 4 (3–5) | < .001 |
| Organization of manuscript | 2 (1–4) | 4 (3–5) | 3 (1–4) | 3 (2–4) | 3 (1–5) | < .001 |
| Interpretation of results | 3 (2–4) | 5 (3–5) | 3 (2–4) | 4 (3–4) | 4 (3–5) | .001 |
| Did reviewer use constructive comments? | 4 (2–5) | 4 (3–5) | 3 (2–4) | 3 (3–4) | 4 (3–5) | .034 |
| Did reviewer use evidence to substantiate comments? | 3 (2–4) | 5 (3–5) | 3 (2–4) | 3 (2–4) | 4 (3–5) | < .001 |
| Overall quality assessment | 3 (2–4) | 4 (3–5) | 3 (2–4) | 3 (3–4) | 4 (3–5) | < .001 |
| Total | 20 (15–30) | 32 (22–38) | 24 (15–30) | 25 (21–30) | 29 (20–38) | .002 |
a For anonymity, reviewers were given a number randomly
b No. of manuscripts chosen for quality assessment of reviews may not equal 10% of total no. of manuscripts reviewed due to exclusion criteria (please see text)
c Data reported as median (range)–data represent sum of scores for 8 questions.
There were 635 individual reviews and recommendations. Overall, the most frequent recommendation was rejection (306 or 48%) followed by acceptance with major revisions (187 or 30%) and acceptance with minor or no revisions (142 or 22%). Table 2 shows the degree of influence of each of the 3 recommendations on the final disposition of the paper for the 2 journals combined as well as separately. For both journals combined, the most influential recommendation was rejection since it was accompanied by a rejection rate of 93%. Second in the degree of influence was the recommendation for acceptance with minor or no revision accompanied by 67% acceptance rate. The recommendation for acceptance with major revisions was accompanied by an overall 40% acceptance rate.
