Study population

The real-world study population was retrospectively recruited from pregnant women presenting with a clinical suspicion of preeclampsia to the obstetrics department, Charité – Universitätsmedizin Berlin, Berlin, Germany, between July 2010 and March 2019. Most of the dataset was previously used in Dröge et al and has been updated and expanded with additional patients (525 additional patients and 837 additional samples).

We included all women aged ≥18 years, with a singleton pregnancy, and a gestational age of ≥20 weeks with signs and symptoms of preeclampsia, available laboratory values for the sFlt-1–to–PlGF ratio, and complete outcome data. “Signs and symptoms” of preeclampsia were defined as elevated blood pressure (systolic blood pressure of ≥140 mm Hg and diastolic blood pressure of ≥90 mm Hg) or preexisting hypertension without concurring proteinuria; proteinuria determined by urine dipstick reading >2+ in 2 separate tests at least 6 hours apart or ≥300 mg/L in 24-hour urine sampling without concurring hypertension; at least 1 preeclampsia-related symptom (headache, visual disturbances, progressive edema and/or excessive weight gain, or upper epigastric pain); abnormalities in specific laboratory values (isolated low platelet count or elevated liver enzymes); fetal abnormalities, such as intrauterine growth restriction (IUGR; <10th percentile) or presenting as small for gestational age (SGA) in sonography; and abnormal values in Doppler sonography of the uterine artery, umbilical artery, or fetal medial cerebral artery.

Per patient data were organized into units of individual days. Of note, 2 measurements, returned from the laboratory on the same day, were consolidated into a single entry for that patient. Moreover, 2 measurements of an identical laboratory test reported on the same day resulted in us using only the most recently performed analysis. In case of multiple measurements at the same visit, we calculated a mean blood pressure.

On average, individual patients had 1.5±0.91 day entries per patient in our database with a median of 1.

Between per-day entries data values, which were at risk of varying, in a clinically relevant manner, from visit to visit (eg, blood pressure, liver enzymes, symptoms related to preeclampsia) were not carried forward and thus were marked as absent or missing.

Entries from the patient history, such as date of birth, height, and medical history, were assumed to be constant and were carried forward to the new per-patient day entries. Finally, in marking up the AOs, if a patient subsequently (in this given pregnancy) suffers an AO, then all visits (day entries) were considered to be potentially indicative of this subsequent AO and thus labeled as such.

In the case of a patient presenting with >1 pregnancy within the given time frame of our investigation, we treated them the same way as a patient who presented without previous knowledge, except for adapting their medical history and biographic information. In case of repeated admissions that did not lead to an AO, these were treated as visits. All women presented without the manifest clinical diagnosis of preeclampsia.

Study outcome

The aim of this study was the assessment of automated ML algorithms as predictive tools in the diagnosis of preeclampsia-associated AOs.

This study’s main target variable was the occurrence of a composite fetal or maternal AO at any time after a visit, up until 14 days after delivery of the neonate, if not specified otherwise. Maternal AOs were cerebral hemorrhage; disseminated intravascular coagulopathy (DIC); pulmonary edema; renal failure; hemolysis, elevated liver enzymes, and low platelet count (HELLP syndrome); eclampsia; and death, whereas AOs in the child were defined as IUGR, SGA, premature delivery because of preeclampsia at ≤34 weeks of gestation, respiratory distress syndrome (RDS), placental abruption, intraventricular hemorrhage (IVH), necrotizing enterocolitis (NEC), or fetal death.

Feature processing

We focused on analyzing routinely available clinical features derived from the current standard of care. These were divided into the broad categories of patient history, laboratory values, ultrasound findings, symptoms at presentation, and previous diagnoses, resulting in a total of 114 features ( Appendix ). Additional values considered were the assessments of sFlt-1 and PlGF and their ratio. sFlt-1 and PlGF were determined using the Elecsys immunoassays for sFlt-1 and PlGF (Roche Diagnostics GmbH, Mannheim, Germany).

To prepare the dataset for ML, we standardized different laboratory measuring systems for a single parameter to adhere to standard units for that parameter. Moreover, we removed any categorical variable via an indicator flag. We analyzed patients on a per-visit basis, thus maintaining biographic variables, such as medical history and diagnoses, occurring before current pregnancy for all subsequent visits. We did not perform any imputation of values and marked all missing data entries with a consistent variable that did not occur within the dataset and which could be detected by the algorithms and treated accordingly.

Furthermore, we classified the sFlt-1 and PlGF measurements and their ratio into reference ranges along known distributions and added these as separate features to the datasets. We did the same for both the umbilical and middle cerebral artery pulsatility indices.

Moreover, we transformed each feature into an additional metric expressing them as a multiple of median (MoM) based on the median obtained from our given dataset.

Dataset splitting

We split the dataset using a train-test approach by applying random sampling of patients moving their entire record into either one of the assessed splits.

Because of limitations regarding the size of our dataset, we chose a 90% to 10% train-test split.

The different classifiers’ performance was compared using a model for a “corrected repeated k-fold cv test” proposed by Bouckaert and Frank. ^,

Algorithms

Thoroughly following best practice guidelines for the development of ML systems, we chose 2 different ML algorithms, random forest (RF) classifier and gradient-boosted tree (GBTree), ^{, ,} for examination of our dataset and focused on methods based on decision tree learning using classification trees. The Appendix and this recent article on medical artificial intelligence (AI) best practices, provides a deeper explanation of the algorithms and the developmental practices followed throughout. These methods provide some limited interpretability for the decision-making of the algorithms and allow clinicians to intuitively understand key aspects of the decision process. We used Shapley values for both methods to provide interpretable metrics for the model’s decision-making.

Throughout hyperparameter tuning and training, we did not manually intervene or tune parameters in any way. Further Information regarding the model creation can be found in the appendix ( Supplemental Table 7 ).

A condensed statement regarding the algorithms setup and structure can be found in the supplementary appendix ( Table 12 ).

Study population characteristics

Continuous variables were tested for normal distribution using the Kolmogorov-Smirnov test.

If both instances of the compared variable were normally distributed, we used the Welch t test for the evaluation of the likelihood that both were drawn from the same distribution; if not, we used the Wilcoxon signed-rank test. For categorical variables, we used the Fisher exact test. Where multiple comparisons were performed, the Bonferroni correction was applied.

Direct biomarker models

To cover different approaches and to draw comparisons with clinical practice, we explored several approaches to derive alternative predictions for our datasets.

• 1.
  

  We used blood pressure as a predictive factor, classifying any women with either a systolic blood pressure ≥140 mm Hg or a diastolic blood pressure ≥90 mm Hg as being predictive of an AO in the future.

  
  
• 2.
  

  We used a cutoff of 38 for the sFlt-1–to–PlGF ratio, which has previously demonstrated a high predictive value for the absence of AO, and used it to classify women as “at risk” if the ratio was >38.

  
  
• 3.
  

  We combined blood pressure threshold, proteinuria, indicated by a urine dipstick measurement of “++” and above or >300 mg/L in 24-hour urine analysis and the sFlt-1–to–PlGF ratio as binary flags combined via a logical odds ratio and analyzed the predictive value of that ensemble.

Cutoff analysis

The basic operation of our tree classifiers was set up to produce a probability for AOs. For standard analysis, we chose a naive cutoff of ≥0.5 to classify outputs as predictive for the probability of an AO.

In a second step, we iterated on the entire range of possible values to derive decision thresholds from the training set and maximize for accuracy, F1 score, and area under the receiver operating characteristics curve (ROCAUC), respectively. Moreover, we applied the individual decision thresholds to the training dataset and analyzed the resulting classification compared with the naive approach.

Model evaluation

We based the evaluation of our model on simple confusion matrix–derived metrics, such as sensitivity, specificity, PPV, and NPV and 4 composite measures for accuracy: the F1 score, threat score, accuracy, and ROCAUC.

Additional data concerning the calibration of the models are available in the Appendix ( Figure 2 ; Supplemental Table 5 ).

Data gathering

Data were gathered manually in an Excel spreadsheet from the hospital information system (SAP Hana, SAP, Walldorf, Germany), ultrasound records (Viewpoint; GE Healthcare, Solingen, Germany), or paper records if none of the others applied.

Ethics approval

All participants consented to the gathering of their data. The study was approved by the ethics committee of the Charité – Universitätsmedizin Berlin, Berlin, Germany.

Study population characteristics

The study population was composed of 1647 patients with a mean maternal age of 33±6 years and a mean gestational age of 34 6/7 weeks (204 gestational days) with a standard deviation of 41 days.

Between patients with an AO and without an AO, there were highly significant ( P <.001) differences in all diastolic blood pressure– and systolic blood pressure–derived features, gestational age in days, height, measurements of blood potassium levels, 24-hour proteinuria, blood urea measurements, headache, or visual disturbance symptoms.

Further features with highly significant differences were no occurrence of symptoms or other pathologic features at the current visit, medical history of hypertension with all features derived from the mean pulsatility index of the uterine arteries, medical history of hypertension, and all sFlt-1–, PlGF-, and sFlt-1–to–PlGF ratio–derived features ( Table 1 ). Full Study population characteristics can be found in the Supplemental Appendix ( Supplemental Table 3 ).

Adverse outcomes

We observed an overall number of 386 (23.4% of patients) AOs, with most being fetal AOs. Of note, 24 children died within 1 week after delivery, 5 children developed IVH, 3 children developed NEC, 12 sustained a placental abruption, 253 neonates were delivered prematurely (<34 0/7 weeks of gestation), and 190 children sustained any form of RDS. The maternal AOs included 1 death during or after delivery, 5 occurrences of lung edema, 8 cases of renal failure, 1 case of DIC, and 33 cases of HELLP syndrome.

The overall number of patients presenting at <34 weeks of gestation was 917 (55.5%) with the remaining 733 patients (44.4%) presenting at ≥34 weeks of gestation.

Of all AOs, 339 cases (87.8%) manifested at <34 weeks of gestation, whereas the remaining 47 cases (12.2%) manifested at ≥34 weeks of gestation.

The fraction of AOs manifesting within 2 weeks after any measurement was 80.8% (312/386) ( Supplemental Table 4 ).

Algorithm performance

We obtained metrics for the 2 naive decision threshold ML models. The GBTree model was performed with a PPV of 81.8%±10%, an NPV of 88.5%±3.5%, a sensitivity of 67.6%±4.3%, a specificity of 94.6%±3%, an overall accuracy of 87.1%±2.8%, an ROCAUC of 0.811±0.029, and an F1 score of 0.737±0.057.

The RF classifier returned an equal PPV (80.8%±9%) and specificity (94.9%±1.9%) while performing slightly inferior on the other available metrics.

Because of their relative similarity, we presented the GBTree figures in the following sections. The results for the RF classifier can be found in the Supplemental Appendix .

Direct biomarker models

We investigated several different approaches to better compare the ML models’ results with clinical approaches represented by standard of care metrics and the sFlt-1–to–PlGF ratio–derived metrics.

• 1.
  

  To understand our dataset, we first examined the predictive value of treating patients with elevated blood pressure as at risk of AOs. This approach performed worse overall than both ML methods across all metrics with a PPV of 32.9%±8.7%, an NPV of 74.5%±6.2%, a sensitivity of 28.9%±7%, a specificity of 78.2%±3.4%, an accuracy of 64.7%±3.2, an F1 score of 0.181±0.046, and an ROCAUC of 0.535±0.037.

  
  
• 2.
  

  Examining the sFlt-1–to–PlGF ratio on its own performed well in terms of NPV (85.5%±3.9%) and sensitivity (69.9%±3.8%), with noticeably lower performance in terms of specificity (67.1%±5.9%) and accuracy (67.8%±4.5%) than the classifiers, with a lower F1 score of 0.372±0.071 and ROCAUC of 0.685±0.036. PPV at 44.5%±9.8% was considerably lower than the ML classifiers. For further information on the patients missed by the biomarker ratio that were classified correctly by the algorithm please review Supplementary Appendix Table 8 . An exemplary subset of patients missed by both the biomarkers and the algorithm can be found in Supplementary Table 9 .

  
  
• 3.
  

  We next combined the blood pressure decision thresholds with proteinuria indicators and the sFlt-1–to–PlGF ratio cutoff. This model performed similarly to the sFlt-1–to–PlGF predictor alone, with high NPV (85.5%±4.1%) and sensitivity (70.4%±4.2%) and comparatively high ROCAUC (0.683±0.034) ( Table 2 ).

  
  
  
  

  Table 2

  
  

  Native statistical classifiers’ (10×10) and clinical models’ performance

  
  

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

  Model	Metric	PPV	NPV	Sensitivity	Specificity	Accuracy	ROCAUC	F1
  GBTree	Average	0.82	0.88	0.68	0.95	0.87	0.81	0.74
  	SD	0.10	0.04	0.04	0.03	0.03	0.03	0.06
  RF	Average	0.81	0.86	0.59	0.95	0.85	0.77	0.68
  	SD	0.09	0.05	0.05	0.02	0.03	0.03	0.05
  Blood pressure cutoffs	Average	0.33	0.74	0.29	0.78	0.65	0.54	0.18
  	SD	0.09	0.06	0.07	0.03	0.04	0.04	0.05
  sFlt-1–to–PlGF cutoff	Average	0.44	0.86	0.70	0.67	0.68	0.69	0.37
  	SD	0.10	0.04	0.04	0.06	0.05	0.04	0.07
  Blood pressure and proteinuria and sFlt-1–to–PlGF cutoff	Average	0.44	0.86	0.70	0.66	0.67	0.68	0.37
  	SD	0.09	0.04	0.04	0.06	0.04	0.03	0.07

  
  

  GBTree , gradient-boosted tree; NPV , negative predictive value; PlGF , placental growth factor; PPV , positive predictive value; RF , Random Forest; ROCAUC , area under the receiver operating characteristic curve; SD , standard deviation; sFlt-1 , soluble fms-like tyrosine kinase-1.

  
  

  Schmidt et al. Machine-learning prediction in preeclampsia. Am J Obstet Gynecol 2022 .
  
  

Maximizing one of the composite metrics (accuracy, F1 score, or ROCAUC) resulted in higher performance across all composite metrics and comparable performance in terms of direct metrics (PPV, NPV, sensitivity, and specificity). Specific values for the mean cutoff by maximized metric can be found in the Appendix ( Supplemental Table 1 ).

The accuracy-optimized GBTree model returned a PPV of 86.9%±7.3%, an NPV of 89.2%±3.6%, a sensitivity of 69.3%±7.1%, a specificity of 95.9%±2.3%, an overall accuracy of 88.7%±2.6%, an ROCAUC of 0.826±0.032, and an F1 score of 0.767±0.049.

Compared with the accuracy optimization, the F1-optimized GBTree model performed with a PPV of 82.8%±6.9%, an NPV of 90.3%±2.2 %, a sensitivity of 72.5%±6.8%, a specificity of 93.8%±4.1%, an accuracy of 88.4%±2.8%, an ROCAUC of 0.832±0.03, and an F1 score of 0.77±0.05.

The ROCAUC-optimized models for the GBTree model returned a PPV of 72.3%±12%, an NPV of 91.9%±2.5%, a sensitivity of 79.4%±4.6%, a specificity of 88.1%±5.9%, an accuracy of 85.7%±3.9%, an ROCAUC of 0.837±0.026, and an F1 score of 0.75±0.066 ( Table 3 ; Figure 1 ).

Mean metrics by cutoff

The graph shows the effect of different classification thresholds on the performance of the model. The mean performance by threshold was based on 10×10 validation, single examples presented with lower opacity for illustration of the effect on individual examples.

Schmidt et al. Machine-learning prediction in preeclampsia. Am J Obstet Gynecol 2022 .

Data concerning the model’s calibration and statistical differences between the optimized threshold models compared with the naive model can be found in the Appendix ( Supplemental Tables 5-7 and Supplemental Figure 2 ).

Interpreting the model

Interpreting the ML models, particularly in the domain of medical AI, is an emerging topic. A larger absolute Shapley value means a parameter typically contributes more to decision-making.

The highest impact features derived by Shapley values were the gestational age (0.725±0.073), measurement of sFlt-1–to–PlGF ratio outside of the 95th percentile (0.237±0.021), sFlt-1–to–PlGF ratio MoM (0.144±0.025), sFlt-1 MoM (0.184±0.023), PlGF deviation from median (0.157±0.028), and height (0.14±0.032) ( Figure 2 ). Concrete values for each features’ mean absolute Shapley values can be found in the appendix ( Supplemental Table 2 ) along with an exemplary illustration of Shapley values’ interpretation for a specific example ( Supplemental Figure 1 ).

Gradient-boosted tree Shapley values single example

The Shapley values were plotted for a single model trained on the dataset. The width represents the number of examples at a specific value, whereas the color represents the relative importance of the feature for a single sample’s prediction.

Schmidt et al. Machine-learning prediction in preeclampsia. Am J Obstet Gynecol 2022 .

Principal findings

We developed a fully automated, easily expandable approach for creating a predictive model out of real-world patient data from a high-risk pregnant population. The resulting companion diagnostic tool was capable of predicting the risk of impending AOs with high accuracy, sensitivity, and PPV. These findings were a preliminary step toward a more integrative diagnostic system for disease in pregnancy.

This was an example of how decision threshold tuning can be used to tune an ML model toward desirable performance characteristics. Moreover, tuning the decision threshold can lead to overfitting and lack of generalization. We addressed this issue by comparing the results across several repetitions through cross-validation and through following best practices for developing medical AI.

Results of the study in context with other observations

The current standard of care for preeclampsia has a low predictive power to detect preeclampsia-related AOs. The discrepancy between the heterogeneity of signs and symptoms for preeclampsia and the severity of the potential complications leads to overdiagnosis and potentially overtherapy. The addition of the sFlt-1–to–PlGF ratio threshold test, which has yet to see widespread adoption, represented a major step toward a more precise detection of the disease and its severe outcomes. ^{, ,} Our ML model, building on top of this and other biomarkers, demonstrated an improvement and other direct biomarker-based clinical decision criteria.

The prediction of preeclampsia employing ML models was recently examined in several scientific papers.

Jhee et al used a multitude of statistical models and found the most effective to be a stochastic gradient boosting method with an ROCAUC of 0.924 while predicting late-onset preeclampsia in a regional cohort of 11,006 pregnant women with a preeclampsia development rate of 4.7%.

Sandström et al used a dataset of 62,562 nulliparous women to evaluate and compare the performance of prespecified variables, backward selection of variables, and a RF classifier. The best performance, with an ROCAUC of 0.68 for the prediction of preeclampsia at <37 weeks of gestation, was obtained by using the prespecified variables model. The RF predictor did not prove superior. In their dataset, 4.4% of women developed preeclampsia.

Marić et al investigated the usage of ML as a tool for early prediction of preeclampsia. They investigated 2 different methods, elastic net and GBTree as algorithms, with the GBTree outperforming the elastic net with an ROCAUC of 0.89. They established the measurements on a retrospective cohort of 16,370 births.

These examples aimed to predict preeclampsia according to the clinical definitions that were currently undergoing substantial change rather than direct AOs. This particular outcome metric has been examined by several different studies, although with very little attention by ML researchers.

As published in the PROGNOSIS study, the sFlt-1–to–PlGF ratio at the cutoff of 38 has a very high NPV to rule out the onset of preeclampsia within 1 week of 99.3% and rule out AOs of 99.5%. The PPV to rule in the disease within 4 weeks was 36.7%. The high NPV was a substantial improvement vs conventional predictive tools and led to a widespread recommendation to use the sFlt-1–to–PlGF ratio as a rule-out test. However, the low PPV has not substantially improved positive prediction and has a lower impact on management decisions in case of increased values. Others have reported a comparable NPV at 98.8% and PPV at 53.5% for the sFlt-1–to–PlGF ratio.

We trained and tested our ML model using a real-world database, which is an expanded and updated version of the database described in Dröge et al. In using an ML approach rather than conventional statistics, we could substantially improve the positive rule-in metrics, such as sensitivity and PPV, which were the main metrics still underperforming in the current standard of care and purely biomarker-based predictive models.

Dröge et al evaluated the application of “conventional” statistics on a previous version of our dataset. Our colleagues could show that multivariate regression analysis yields markedly better results than sFlt-1–to–PlGF ratio or blood pressure and proteinuria alone. The area under the curve for the full model proved even higher than the one presented in this paper with 0.887 while retaining high positive and negative predictive power (specificity, 87.3%; sensitivity, 80%; NPV, 90.2%; PPV, 75%). This signified the validity of more traditional statistical approaches. However, creating it required considerably more effort and hands-on corrections, which is a key advantage of our methodology. Our automated pipeline was capable of generating noninferior models with remarkably less work overhead.

These examples showed the potential of ML models to match conventional statistical apporaches and outperform the current standard of care. The application for the prediction of AOs, which our paper presented a first step toward, has yet to be thoroughly investigated.

Moreover, although we acknowledge that screening for preeclampsia is recommended in the first trimester in low-risk women, we focused on short-term prediction of women presenting with signs and symptoms in later stages of pregnancy because of the higher intrinsic risk of our study population.

Clinical implications

Integrating ML-based algorithms in decision support tools provides a veritable improvement in the clinician’s ability to make informed decisions about their patients’ health status and appropriate treatments.

Using a formulated and established approach for the development of ML in healthcare, our findings were a relevant step toward a structured pipeline that connects medicine to modern ML practices.

Furthermore, we showed how a step toward a more interpretable ML model can be made. By using Shapley values, we gained limited insight into the weights associated with certain features as assigned by the algorithms.

This can provide a step toward truly interpretable models, especially with the adoption of the General Data Protection Regulation in the European Union, requiring “meaningful information about the logic involved” in automated decision processes.

Research implications

The presented research provided several different avenues for further investigation in both the clinical and theoretical fields.

First, the set of features we used to derive the ML model should be further investigated for additions, such as other biomarkers or exclusions, based on the statistics we provided for measuring the features’ impact on the decision made by the ML models.

Second, the research should be verified in other populations than the one presented in this paper, especially regarding ethnic composition and preliminary risk of preeclampsia.

Another topic for follow-up could be the application in different contexts, especially concerning modern capabilities of data collection, such as home monitoring and/or high-resolution time series data obtained in a clinical setting.

In addition, the topic of postpartum preeclampsia, which was not subject in our study, could prove worthy of further research using the presented methods.

Strengths and limitations

Our approach presented a fully automated system that was capable of incorporating considerably more clinical parameters than traditional approaches. It required very little interaction, handled missing features without the need for data imputation, and was, to a limited extent, tunable for specific characteristics.

The most significant limitation of our study was, from an ML perspective, the small size of the dataset, which posed the risk of overfitting the obtained results.

Second, the underlying patient collective represents a high-risk population, from a single hospital group, which limited the generalization to low-risk pregnant women. Moreover, we admit the potential for intervention bias, which could contribute to the relatively low predictive ability of the serum markers, as elevated serum levels would lead to increased attention and possible intervention.

Third, we were bound to use manual data gathering. Direct access to the clinical information system and its output would most likely increase the results’ reliability.

Fourth, there was an inherent risk for ML approaches to obtain patterns specific to a certain dataset and generate an “overfitted” model. High dimensionality and limited understanding of the decision-making processes of a fitted model represented significant obstacles in detecting erroneous pattern matching.

The need for data availability and interoperability was a key concern throughout all fields of medical research and posed an especially important challenge for the development of medical AI.

These limitations were strongly mitigated by the depth of our methodologic approach and the scale and international nature of the population served by the treating clinic.

Conclusion

We presented a working, automated, and scalable system for the prediction of AOs in high-risk pregnant women. The presented ML models performed with high accuracy and overall reliability and could be easily deployed to a clinical environment as a companion diagnostic tool for pregnant women and provide valuable decision support for physicians in both clinical and private practices.

Using ML to combine patient history with basic laboratory results, in an sFlt-1–to–PlGF ratio decision paradigm, greatly improved the rule-in predictive metrics while maintaining excellent performance on rule-out criteria.

Although the application of these methods was still in the research phase, the results, concerning the current extent of research performed, showed definite promise for a clinical decision support tool.