Study design

From 2008-2011, we assembled a cohort of women and their neonates born in any of the 25 hospitals of the Eunice Kennedy Shriver National Institute of Child Health and Human Development Maternal-Fetal Medicine Units Network. The Assessment of Perinatal Excellence study was designed to develop quality measures for intrapartum obstetrical care. This study was approved by the institutional review board at each participating institution under a waiver of informed consent.

Patients eligible for data collection were those who delivered within the institution, were at least 23 weeks of gestation, and had a live fetus on admission. Data were collected on eligible patients if they delivered during the 24-hour period of selected days during a 3-year period (March 2008 – February 2011). Days were chosen via computer-generated random selection. To avoid overrepresentation of patients from larger hospitals, we selected one third of days at hospitals with annual delivery volumes from 2000-7000 and up to one sixth of days at hospitals with annual deliveries >7000. The randomization scheme was stratified by weekdays, weekends, and holidays and generated separately for each hospital. On selected days, the labor and delivery logbook at each participating center was screened to identify all eligible women. The medical records of all eligible women and their newborns were abstracted by trained and certified research personnel at the hospital and entered into a World Wide Web–based data entry system. Data recorded included demographic characteristics, details of the medical and obstetrical history, information about intrapartum and postpartum events, and patients’ race and ethnicity as reported in the chart. Maternal data were collected until discharge and neonatal data were collected up until discharge or until 120 days of age, whichever came first.

Feasibility and quality of data collection were ensured by several mechanisms. First, prior to selecting final data fields and forms, a 2-week pilot study took place to evaluate the data collection process, quality of the data, and frequency of missing data. Based on the information gathered during this pilot phase, final data fields were selected and forms revised. All data were subjected to ongoing data edits to ensure accuracy.

Primary outcomes

An initial determination of the primary obstetric outcomes of interest was made via expert consensus, obtained during meetings of members of the Maternal-Fetal Medicine Units Steering Committee and an external advisory committee convened specifically for this project (Acknowledgments). Based on input from these committees, 5 primary outcomes were chosen because they represented different domains of obstetric complications, were clinically meaningful, could be affected by differences in clinical care, were ascertainable from medical records, and potentially occurred with sufficient frequency to allow valid institutional comparisons: venous thromboembolism, postpartum hemorrhage, peripartum infection, severe perineal laceration, and a composite neonatal adverse outcome. Venous thromboembolism was defined as occurrence of either a deep venous thrombosis diagnosed by duplex Doppler or a pulmonary embolism diagnosed by computed tomography or ventilation-perfusion lung scan. Postpartum hemorrhage was defined as occurrence of any of the following: an estimated blood loss ≥1500 mL at delivery or the immediate postpartum period, a blood transfusion, or a hysterectomy for hemorrhage, placenta accreta, or atony. Peripartum infection was defined as occurrence of any of the following: chorioamnionitis, endometritis, wound cellulitis requiring antibiotics, wound reopened for fluid collection or infection, or wound dehiscence during the delivery hospitalization. Severe perineal laceration was defined as the occurrence of a third- or fourth-degree perineal laceration, was restricted to women with vaginal singleton deliveries with no shoulder dystocia or placenta previa, and was stratified by spontaneous, vacuum, or forceps delivery. The composite neonatal adverse outcome was defined as occurrence of any of the following restricted to term (≥37 weeks of gestation), nonanomalous singleton infants: neonatal stay longer than maternal stay by ≥3 calendar days, 5-minute Apgar score <4, skeletal fracture other than of the clavicle, facial nerve palsy, brachial plexus palsy, subgaleal hemorrhage, ventilator support, hypoxic ischemic encephalopathy, stillbirth after hospital admission, or neonatal death. We chose to include only term infants because morbidity prior to term is strongly related to prematurity-related complications and not necessarily differences in health care and thus inclusion is not as good a candidate for an obstetric quality measure. Additional details regarding the definitions of these outcomes and relevant denominators can be found in the Appendix ; Supplementary Table 1 .

Statistical analyses

At each institution, the unadjusted frequencies of adverse outcomes, with 95% confidence intervals (CIs), were calculated and were compared using the χ ² test. The analysis was then directed at assessing which patient characteristics were significantly associated with the chosen outcomes. We chose not to examine hospital characteristics such as volume or teaching status in the model as they are not inherently related to the patient and should not be included in a risk-adjustment model. Patient characteristics eligible for multivariable models were selected a priori based on whether they could plausibly be associated with the outcome. While inclusion of the same risk factors in all models would be convenient, requiring this condition could lead to loss of precision of the risk-adjustment models that were developed. As our goal was to have the best possible risk-adjustment models, we included as potential factors for each model the most relevant and plausible risk factors. Prior to multivariable analysis, the possibility of colinearity among patient characteristics was assessed. Continuous variables were first assessed to determine whether their association with each outcome was linear, by assessing the linearity of the log (odds), using a locally weighted scatterplot smoothing technique. When there was evidence of nonlinearity, we included both linear and quadratic terms. Model selection was based on creating derivation and validation datasets using a k-fold cross-validation approach in which the cohort was randomly divided into 10 equal parts and logistic regression models, using backward selection, were generated utilizing every possible combination of 9 of the 10 sets. Variables with P < .05 were retained, and each of the 10 subsamples was used for validation. The C statistic was computed to assess each model’s predictive ability (discrimination). Only those variables that were present in the logistic regression model with the highest C statistic and also were present in at least 8 of the 10 k-fold logistic regression models were chosen for the final multivariable model that included the entire dataset. Because assessment of the Hosmer-Lemeshow test statistic ( P value) is not recommended for datasets as large as ours, model fit was assessed from graphical displays of the observed and expected number of patients within each partition of the Hosmer-Lemeshow test.

The final multivariable models were then used to estimate hospitals’ expected outcome frequencies. To estimate a hospital’s expected outcome frequency, which is the hospital’s outcome frequency that would be expected given the characteristics of their patients, the predicted outcome probability was estimated for each patient and then all patient probabilities within the same hospital were averaged (see eStatistics text). These expected outcome frequencies were used to calculate an observed (unadjusted) to expected ratio (OER). Bootstrapping was performed on 1000 samples with replacement to estimate 99% CI around the OER and identify the hospitals that were significantly different from an OER of 1.0. OERs can be interpreted as such: if the ratio is <1.0, the hospital has fewer adverse outcomes than expected; if the ratio = 1.0, the hospital has as many adverse outcomes as expected; and if the ratio is >1.0, the hospital has more adverse outcomes than expected. Because we were estimating individual hospital frequencies, the primary models did not adjust for hospital; however, regressions accounting for patient clustering within a hospital (ie, adding hospital as a fixed effect to the logistic model or as a random effect to a hierarchical model) were performed to evaluate whether either adjustment altered the strength and precision of the estimated odds ratios (ORs) for the patient characteristics.

For each outcome, hospitals were ranked according to their unadjusted frequency and reranked according to their adjusted frequency, and Kendall coefficient of concordance was used to assess the degree to which these rankings were similar. Correlations of hospital-adjusted frequencies for each pair of outcomes were tested using Spearman rank correlation.

SAS software (SAS Institute, Cary, NC) was used for the analyses. All tests were 2-tailed. P < .01 was used to define statistical significance and 99% CIs were estimated when directly testing a hypothesis, ie, correlations between outcomes, concordance between unadjusted and adjusted ranks, and to identify hospital outliers. P < .05 and 95% CIs were estimated for model building and more descriptive analyses.