Why use prognostic models?

The ultimate goal of risk-based management is to allow for timely diagnosis and prognosis and consequently more effective management. Based on patients’ risks, interventions can be applied to those that potentially benefit most, thereby improving patient outcomes while saving costs and avoiding unnecessary burden by refraining from treatment for individuals unlikely to benefit from a certain intervention. Here, prognostic models can aid in several ways. They may serve as an alert (eg, when a woman or newborn needs immediate medical care), be used for individual decision making (eg, to choose an alternative treatment or refrain from treatment when the chances of success or improvement are low), aid in organizational planning (eg, availability of staff and the operating theater when the risk of an operative delivery is high), or allow for individualized counseling of patients (eg, in the decision of a pregnant woman to opt for external cephalic version). We will first describe the currently available models in obstetrics for potential use.

Available prognostic models in obstetrics

As there was no overview of prognostic models available for potential use, we undertook a systematic review of the literature on this topic adhering to the principles in the PRISMA statement.

We searched MEDLINE through PubMed on July 1, 2012, without language or publication date restrictions to identify papers reporting on the development of a prognostic model in obstetrics. The search strategy was based on terms related to women, pregnancy, and obstetrical topics combined with sensitive and specific methodological filters, allowing efficient identification of publications on prognostic models ( Appendix ; Supplementary Tables 1 and 2 ). We defined a prognostic model as a model that could be used to estimate risks for individual patients, or to distinguish groups of patients at different risks, based on ≥3 predictor variables. Papers were excluded if the described model was to be used for diagnosis of a current condition rather than for predicting a future outcome, or the predicted outcome was outside the field of obstetrics, defined as an outcome concerning the pregnant, laboring, or postpartum woman immediately after delivery, or the fetus (eg, fetal growth restriction), or the neonate immediately after birth (eg, birthweight). Eligible papers were selected by 2 reviewers (C.E.K. and E.S.). Titles and abstracts of all papers identified by the search were scrutinized for eligibility. After 300 abstracts were screened in duplicate, which showed good agreement between both reviewers (kappa statistic = 0.71, indicating that 98.7% of papers were scored the same by both reviewers), the remaining were assessed by 1 reviewer. In case of doubt about eligibility, the paper was discussed with the other reviewer to accomplish a joint decision. Once selected, both reviewers examined the full text papers to see whether they met the inclusion criteria. Disagreements about inclusion at any stage of the selection process were resolved by consensus.

From 10,152 citations, a total of 177 papers met inclusion criteria and described the development of ≥1 obstetric prognostic model ( Figure 1 ). Overall, we identified 263 models for 40 different outcomes. The oldest paper identified was published in 1976. Since then, the number of available papers and prognostic models increased markedly ( Figure 2 ). These findings are consistent with those reported elsewhere for other clinical areas. The marked increase in the number of available obstetrical prognostic models indicates that clinicians and researchers are increasingly interested in risk-based medicine or personalized medicine.

Selection of studies for inclusion in systematic review

Kleinrouweler. Prognostic models in obstetrics. Am J Obstet Gynecol 2016 .

Number of papers describing prognostic models, available models, and external validation studies

One paper (describing 2 models) was published online in 2012 and in print in 2013. With advancing period, total number of models increased more than total number of papers describing these models, so more papers describing >1 model were published. Number of external validation studies does not increase as markedly as number of published prediction models.

Kleinrouweler. Prognostic models in obstetrics. Am J Obstet Gynecol 2016 .

Of the 40 different outcomes that were predicted, there were 15 outcomes for which only 1 model was developed, 14 outcomes for which 2 or 3 models were identified, 6 outcomes for which between 4-10 models were available, and 5 outcomes with ≥10 published models. The 5 most frequently predicted outcomes were preeclampsia (n = 69), preterm delivery (n = 63), mode of delivery (n = 22), gestational hypertension (n = 11), and small-for-gestational-age neonates (n = 10). Among these 5 outcomes, many more models were developed for predicting preeclampsia and preterm delivery than for any other outcome. Both preeclampsia and preterm delivery are highly prevalent conditions in obstetrics (up to, and just over, an incidence of 10%, respectively) and are major causes of adverse outcomes. The prevalence and clinical importance of these conditions and the potential benefits of early (preventive) treatment (aspirin for prevention of preeclampsia and progestagens, pessary or cerclage for prevention of preterm delivery) and organization of care (antenatal corticosteroids, intrauterine transfer to centers with neonatal intensive care facilities) may explain the large number of models developed for these outcomes.

Available prognostic models in obstetrics

As there was no overview of prognostic models available for potential use, we undertook a systematic review of the literature on this topic adhering to the principles in the PRISMA statement.

We searched MEDLINE through PubMed on July 1, 2012, without language or publication date restrictions to identify papers reporting on the development of a prognostic model in obstetrics. The search strategy was based on terms related to women, pregnancy, and obstetrical topics combined with sensitive and specific methodological filters, allowing efficient identification of publications on prognostic models ( Appendix ; Supplementary Tables 1 and 2 ). We defined a prognostic model as a model that could be used to estimate risks for individual patients, or to distinguish groups of patients at different risks, based on ≥3 predictor variables. Papers were excluded if the described model was to be used for diagnosis of a current condition rather than for predicting a future outcome, or the predicted outcome was outside the field of obstetrics, defined as an outcome concerning the pregnant, laboring, or postpartum woman immediately after delivery, or the fetus (eg, fetal growth restriction), or the neonate immediately after birth (eg, birthweight). Eligible papers were selected by 2 reviewers (C.E.K. and E.S.). Titles and abstracts of all papers identified by the search were scrutinized for eligibility. After 300 abstracts were screened in duplicate, which showed good agreement between both reviewers (kappa statistic = 0.71, indicating that 98.7% of papers were scored the same by both reviewers), the remaining were assessed by 1 reviewer. In case of doubt about eligibility, the paper was discussed with the other reviewer to accomplish a joint decision. Once selected, both reviewers examined the full text papers to see whether they met the inclusion criteria. Disagreements about inclusion at any stage of the selection process were resolved by consensus.

From 10,152 citations, a total of 177 papers met inclusion criteria and described the development of ≥1 obstetric prognostic model ( Figure 1 ). Overall, we identified 263 models for 40 different outcomes. The oldest paper identified was published in 1976. Since then, the number of available papers and prognostic models increased markedly ( Figure 2 ). These findings are consistent with those reported elsewhere for other clinical areas. The marked increase in the number of available obstetrical prognostic models indicates that clinicians and researchers are increasingly interested in risk-based medicine or personalized medicine.

Selection of studies for inclusion in systematic review

Kleinrouweler. Prognostic models in obstetrics. Am J Obstet Gynecol 2016 .

Number of papers describing prognostic models, available models, and external validation studies

One paper (describing 2 models) was published online in 2012 and in print in 2013. With advancing period, total number of models increased more than total number of papers describing these models, so more papers describing >1 model were published. Number of external validation studies does not increase as markedly as number of published prediction models.

Kleinrouweler. Prognostic models in obstetrics. Am J Obstet Gynecol 2016 .

Of the 40 different outcomes that were predicted, there were 15 outcomes for which only 1 model was developed, 14 outcomes for which 2 or 3 models were identified, 6 outcomes for which between 4-10 models were available, and 5 outcomes with ≥10 published models. The 5 most frequently predicted outcomes were preeclampsia (n = 69), preterm delivery (n = 63), mode of delivery (n = 22), gestational hypertension (n = 11), and small-for-gestational-age neonates (n = 10). Among these 5 outcomes, many more models were developed for predicting preeclampsia and preterm delivery than for any other outcome. Both preeclampsia and preterm delivery are highly prevalent conditions in obstetrics (up to, and just over, an incidence of 10%, respectively) and are major causes of adverse outcomes. The prevalence and clinical importance of these conditions and the potential benefits of early (preventive) treatment (aspirin for prevention of preeclampsia and progestagens, pessary or cerclage for prevention of preterm delivery) and organization of care (antenatal corticosteroids, intrauterine transfer to centers with neonatal intensive care facilities) may explain the large number of models developed for these outcomes.

Important aspects to consider when appraising models

Table 1 describes some of the important concepts in prognostic model studies. Papers describing a prognostic model should report these items that allow for assessment of model performance and applicability, as well as describe a clear definition of the predictors and outcome, details of the population studied, study design (eg, cohort or case control), sample size, and statistical methods including selection of predictors and handling of missing data. Each model included in the systematic review was thoroughly assessed for methodological development and validation as well as reporting, with the use of a checklist based on the CHARMS checklist and a previously conducted systematic review. Supplementary Table 2 gives an overview of all included models and details of their performance, validity, and applicability.

Whether the identified models in obstetrics reported model performance (calibration and discrimination), presented the model such that it could be used by others, gave guidance for clinical use, and were internally or externally validated (or both) is also shown in Table 2 . Of the 263 identified models, 57 (21.7%) were internally validated and only 23 (8.7%) were externally validated ( Figure 2 ). Details of model performance, either apparent or at internal or external validation, included calibration of only 46 models (17.5%), details of discrimination for 165 models (62.7%), and both calibration and discrimination were presented for 45 models (17.1%). A prognostic formula, rule or score that could be used by others was reported for 164 models (62.4%) and guidance for clinical use was discussed for 29 (11.0%).

For predicting preeclampsia and preterm delivery many models have been developed, but only 7.2% (5/69) and 6.3% (4/63), respectively, have been externally validated. Overall, model performance was lower at external validation than at apparent or internal validation. High discrimination (area under the receiver operating characteristic curve [AUC] >0.90) either at apparent or internal validation was observed in the development phase of 25% and 24% of models for which discrimination was presented, respectively. For preterm delivery, AUCs of models at external validation ranged between 0.65–0.72. For preeclampsia, AUCs were between 0.70–0.85 (all models for late preeclampsia). For both outcomes we found that recently developed models did not have better performance than already existing models ( Supplementary Table 2 ).

Why are existing models not yet applicable and not being used?

Despite the availability of many models for various outcomes, we are not aware of any (recent) obstetrical models that have found their way into routine practice. The reasons for this may be multiple. First, clinicians may be in doubt on whether to rely on probabilities provided by these models (ie, face validity), because models may not include well-known predictors of the outcome. Additionally, in obstetrics there tends to be an inverse relation between outcomes of mother and baby, eg, early delivery to benefit the mother might compromise baby’s outcome due to prematurity, so combining models developed on different data sets for maternal and perinatal outcomes may lower face validity. Another related issue, sometimes described as treatment paradox, can corrupt face validity when models are appropriately developed within the same data set. Predictors of the outcome or interventions that may benefit the mother but at the same time compromise the baby’s health or vice versa, leave the clinician in doubt on whether to rely on probabilities provided by these models. Second, individuals often rely on simple heuristics, ie, cognitive processes, conscious or unconscious, that ignore part of the information, which hampers incorporation of prognostic models into clinical practice. Third, prognostic models are often too complex for daily use in clinical settings without computer support (although the introduction of computerized patient records will clearly facilitate their application in routine care). Fourth, preventive treatment for the outcome that is predicted may not exist, so clinicians may prefer expectant management and treat the patient when the disease eventually develops instead of using a model. Fifth, many prognostic models have not been validated in other populations, meaning that their generalizability is unclear. And, finally, the reporting and methodological quality may be questionable or unclear, despite (recommended) methods for development of prognostic models that are well described and available in commonly used statistical software.

There are several steps to be taken between the development of a prognostic model and its use in practice. Firstly, one should question whether the implementation of a prognostic model with acceptable performance is likely to improve patient care, decision making, patient outcomes, counseling, or organization of care. Most authors of the papers identified by our systematic review described in their introduction that the possibility of accurate risk estimation for a variety of conditions would potentially have huge advantages. However, in the discussion it was usually not described how the prognostic model should be used or what was defined as being low or high risk. Secondly, a prognostic model should be developed using a sound methodological approach and its performance combined with ease of application should indicate whether the models warrants further investigation and validation. Interestingly, while reading the full-text papers, we observed that many reported that the addition of biomarkers (eg, cervical length, fetal fibronectin, serum protein levels) to characteristics of maternal and obstetric history (age, parity, previous conditions) and routine examinations (body mass index, blood pressure) often improved model discrimination up to an acceptable performance. However, if acceptable performance can be achieved with the use of only readily available variables this could make a model easier (and therefore perhaps more likely) to be used in practice. Thirdly, new or existing models should be externally validated, ideally by independent investigators, and compared to competing models, and, if necessary, updated. Less than 10% of the identified models in obstetrics have been externally validated, either in the same paper or by an independent research group. Additionally, only 164 models (62.4%) were presented in a manner for others to use, meaning that either external validation or clinical use would be possible. A fourth step after validation is to investigate whether using the prognostic model actually improves patient care: whether patient outcomes are better when changes in clinical management are made based on, or are supported by, the prognostic information provided by the model. This can be studied, for example, in a randomized trial that compares ≥1 treatment strategy guided by the prognostic model with care as usual (without the model) for relevant patient outcomes. Only 1 paper in our review discussed the issue of conducting a randomized trial to evaluate the potential clinical benefits of use of the model. For all others models, impact studies–or evidence of their initiation–were lacking. In the phase of assessing model impact it should be clear how the model should be used in practice and ideally there should be a recommendation for management of women at risks higher and lower than a certain threshold. Only a tenth of the models in this review discussed guidance for future use.

Comment

The increasing number of prognostic models for the same outcome along with the dearth of (independent) external validation studies and, more importantly, studies evaluating the clinical impact of using the prognostic model, seems to indicate that researchers in the field of obstetrics fail to appreciate the steps required in the introduction of a new prognostic model. In addition, the realization that almost 40% of papers failed to present their model in a format that could be used by others (eg, for external validation) could indicate that developers of prognostic models might not be aware of the necessity of (multiple) external validation. Furthermore, researchers seem not to critically question the consequences of developing another model along existing models for the same outcome. It has even been suggested that the increase in prognostic models can be partly attributed to the simplicity of publishing yet another paper, simply for sake of publication rather than its potential use in clinical practice. We suggest that before efforts to develop a new model are undertaken, systematic reviews should be carried out to identify and validate existing models with careful consideration to decide whether to develop a new model or update an existing model. Afterwards, the most valid, best-performing models should be studied in clinical practice in so-called impact studies to investigate influence on patient outcomes.

The usefulness of any prognostic model is predicated on full and transparent reporting of how the model was developed and validated. If key details are not reported, including the actual prognostic model (for which our review finds that 38% were insufficiently reported), then deciding whether a prognostic model has potential for practice is difficult. Systematic reviews of prognostic models in other areas of medicine have described inappropriate methodology and reporting, similar to our findings. Consensus-based guidelines have recently been published to assist authors, readers, reviewers, and journal editors on issues to report when developing or validating a prognostic model. When a model is already published, contacting the developers with a request for cooperation or additional information may help those who wish to further evaluate a model.

We have not described any assessment of the methodological conduct of the developed models and the quality of reporting of key methodological items. Although methodological quality of a model development study is–arguably–of less importance when a model shows good performance at external validation, the likelihood of developing a generalizable model (which is not overfitted to the data on which the model is developed) is higher when recommended statistical methods have been used.

Based on the large amount of prognostic modeling papers, clinicians and researchers in obstetrics seem to be open to (the use of) prognostic models, but there should be realization that it takes far more than just developing a model before patients can benefit from it. Although most authors of the papers identified by our systematic review described that accurate risk estimation would potentially have huge advantages, it remains unclear if using any of the identified models will truly impact clinical practice and contribute to improvement of patient care. Thus, at this point, we cannot recommend the routine use of any of the models. Given the potential benefits for timely prognostication and effective management, it seems unfortunate that there is such a low number of applied models. The Framingham risk score for future occurrence of cardiovascular disease, on which the decision to start preventive interventions is based, is an example of a well-studied and widely implemented prognostic model. Consequently, further investigation of model validity and impact is important and should be undertaken.

In conclusion, in obstetrics many prognostic models are developed but there is relatively little evidence about their performance, impact, and usefulness in clinical practice. New efforts in this context–or outside obstetrics–should be directed towards evaluating the performance and impact of these existing models rather than developing new ones.