Prediction Models for COVID-19 Diagnosis & Prognosis - PDF | BMJ

Research Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal Laure Wynants,1,2 Ben Van Calster,2,3 Gary S Collins,4,5 Richard D Riley,6 Georg H...

Research Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal Laure Wynants,1,2 Ben Van Calster,2,3 Gary S Collins,4,5 Richard D Riley,6 Georg Heinze,7 BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. Ewoud Schuit,8,9 Elena Albu,2 Banafsheh Arshi,1 Vanesa Bellou,10 Marc M J Bonten,8,11 Darren L Dahly,12,13 Johanna A Damen,8,9 Thomas P A Debray,8,14 Valentijn M T de Jong,8,9 Maarten De Vos,2,15 Paula Dhiman,4,5 Joie Ensor,6 Shan Gao,2 Maria C Haller,7,16 Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. Michael O Harhay,17,18 Liesbet Henckaerts,19,20 Pauline Heus,8,9 Jeroen Hoogland,8 Mohammed Hudda,21 Kevin Jenniskens,8,9 Michael Kammer,7,22 Nina Kreuzberger,23 Anna Lohmann,24 Brooke Levis,6 Kim Luijken,24 Jie Ma,5 Glen P Martin,25 David J McLernon,26 Constanza L Andaur Navarro,8,9 Johannes B Reitsma,8,9 Jamie C Sergeant,27,28 Chunhu Shi,29 Nicole Skoetz,22 Luc J M Smits,1 Kym I E Snell,6 Matthew Sperrin,30 René Spijker,8,9,31 Ewout W Steyerberg,3 Toshihiko Takada,8,32 Ioanna Tzoulaki,10,33 Sander M J van Kuijk,34 Bas C T van Bussel,1,35 Iwan C C van der Horst,35 Kelly Reeve,36 Florien S van Royen,8 Jan Y Verbakel,37,38 Christine Wallisch,7,39,40 Jack Wilkinson,24 Robert Wolff,41 Lotty Hooft,8,9 Karel G M Moons,8,9 Maarten van Smeden8 For numbered affiliations see Abstract Data extraction end of the article Objective At least two authors independently extracted data using Correspondence to: L Wynants To review and appraise the validity and usefulness the CHARMS (critical appraisal and data extraction for laure.wynants@ of published and preprint reports of prediction systematic reviews of prediction modelling studies) maastrichtuniversity.nl (ORCID 0000-0002-3037-122X) models for prognosis of patients with covid-19, and checklist; risk of bias was assessed using PROBAST Additional material is published for detecting people in the general population at (prediction model risk of bias assessment tool). online only. To view please visit increased risk of covid-19 infection or being admitted Results the journal online. to hospital or dying with the disease. 126 978 titles were screened, and 412 studies Cite this as: BMJ 2020;369:m1328 http://dx.doi.org/10.1136/bmj. Design describing 731 new prediction models or validations m1328 Living systematic review and critical appraisal by the were included. Of these 731, 125 were diagnostic Originally accepted: covid-PRECISE (Precise Risk Estimation to optimise models (including 75 based on medical imaging) 31 March 2020 covid-19 Care for Infected or Suspected patients in and the remaining 606 were prognostic models Final version accepted: diverse sEttings) group. for either identifying those at risk of covid-19 in 17 July 2022 Data sources the general population (13 models) or predicting PubMed and Embase through Ovid, up to 17 February diverse outcomes in those individuals with confirmed 2021, supplemented with arXiv, medRxiv, and bioRxiv covid-19 (593 models). Owing to the widespread up to 5 May 2020. availability of diagnostic testing capacity after the summer of 2020, this living review has now focused Study selection on the prognostic models. Of these, 29 had low risk Studies that developed or validated a multivariable of bias, 32 had unclear risk of bias, and 545 had covid-19 related prediction model. high risk of bias. The most common causes for high risk of bias were inadequate sample sizes (n=408, 67%) and inappropriate or incomplete evaluation of What is already known on this topic model performance (n=338, 56%). 381 models were Recurrent peaks in covid-19 incidence have put a strain on healthcare systems newly developed, and 225 were external validations worldwide; a need exists for efficient early risk stratification in the general of existing models. The reported C indexes varied population, and for prognosis of covid-19 in patients with confirmed disease between 0.77 and 0.93 in development studies Viral nucleic acid testing, chest computed tomography imaging, and antigen with low risk of bias, and between 0.56 and 0.78 in tests are standard methods for diagnosing covid-19, and their availability has external validations with low risk of bias. The Qcovid made covid-19 diagnostic models less relevant models, the PRIEST score, Carr’s model, the ISARIC4C Deterioration model, and the Xie model showed Earlier updates of this living review could not find models at low risk of bias adequate predictive performance in studies at low risk What this study adds of bias. Details on all reviewed models are publicly available at https://www.covprecise.org/. Of models with a low risk of bias, four identify patients at risk in the general population; one assists in patient triage at the emergency department; and three Conclusion estimate prognosis in patients admitted to hospital with covid-19 Prediction models for covid-19 entered the academic literature to support medical decision making at Calibration of these models is likely to vary over time and across settings unprecedented speed and in large numbers. Most There is an oversupply of models and external validations at high risk of bias, published prediction model studies were poorly raising concern that predictions could be unreliable when these models are reported and at high risk of bias such that their applied in dailly practice reported predictive performances are probably the bmj | BMJ 2020;369:m1328 | doi: 10.1136/bmj.m1328 1  Research optimistic. Models with low risk of bias should The outbreak of covid-19 was accompanied by be validated before clinical implementation, a surge of scientific evidence.9 The speed with preferably through collaborative efforts to also which evidence about covid-19 has accumulated is allow an investigation of the heterogeneity in their unprecedented. To provide an overview of available performance across various populations and settings. prediction models, a living systematic review, with BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. Methodological guidance, as provided in this paper, periodic updates, was conducted by the international should be followed because unreliable predictions covid-PRECISE (Precise Risk Estimation to optimise could cause more harm than benefit in guiding clinical covid-19 Care for Infected or Suspected patients in decisions. Finally, prediction modellers should adhere Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. diverse sEttings; https://www.covprecise.org/) group to the TRIPOD (transparent reporting of a multivariable in collaboration with the Cochrane Prognosis Methods prediction model for individual prognosis or Group. Initially, the review included diagnostic and diagnosis) reporting guideline. prognostic models. Owing to the current availability Systematic review registration of testing for covid-19 infections, we restricted the Protocol https://osf.io/ehc47/, registration https:// focus to prognostic models in this new update. Hence osf.io/wy245. our aim was to systematically review and critically Readers’ note appraise available prognostic models for detecting This article is the final version of a living systematic people in the general population at increased risk review that has been updated over the past two years of covid-19 infection or being admitted to hospital to reflect emerging evidence. This version is update or dying with the disease, and models to predict the 4 of the original article published on 7 April 2020 prognosis or course of infection in patients with a (BMJ 2020;369:m1328). Previous updates can be covid-19 diagnosis. We included all prognostic model found as data supplements (https://www.bmj.com/ development and external validation studies. content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update Methods number and date of access for clarity. We searched the publicly available, continuously updated publication list of the covid-19 living Introduction systematic review.10 We validated whether the list is The novel coronavirus disease 2019 (covid-19) fit for purpose (online supplementary material) and presents an important threat to global health. Since further supplemented it with studies on covid-19 the outbreak in early December 2019 in the Hubei retrieved from arXiv. The online supplementary province of the People’s Republic of China, the number material presents the search strings. We included of patients confirmed to have the disease has exceeded studies if they developed or validated a multivariable 500 million as the disease spread globally, and the model or scoring system, based on individual number of people infected is probably much higher. participant level data, to predict any covid-19 related More than 6 million people have died from covid-19 outcome. These models included prognostic models (up to 24 May 2022).1 Despite public health responses to predict the course of infection in patients with aimed at containing the disease and delaying the covid-19; and prediction models to identify people spread, countries have been confronted with repeated in the general population at risk of covid-19 infection surges disrupting health services2-6 and society at or at risk of being admitted to hospital or dying with large. More recent outbreaks of the omicron variant led the disease. Diagnostic models to predict the presence to important increases in the demand for test capacity, or severity of covid-19 in patients with suspected hospital beds, and medical equipment, while medical infection were included up to update 3 only, which can staff members also increasingly became infected be found in the data supplements. themselves.6 While many national governments have We searched the database repeatedly up to 17 now put an end to covid-19 restrictions, scientists warn February 2021 (supplementary table 1). As of the third that endemic circulation of SARS-CoV-2, perhaps with update (search date 1 July 2020), we only include peer seasonal epidemic peaks, is likely to have a continued reviewed articles (indexed in PubMed and Embase important disease burden.7 In addition, virus through Ovid). Preprints (from bioRxiv, medRxiv, and mutations can be unpredictable, and lack of effective arXiv) that were already included in previous updates surveillance or adequate response could enable the of the systematic review remained included in the emergence of new epidemic or pandemic covid-19 analysis. Reassessment took place after publication patterns.7 8 To mitigate the burden on the healthcare of a preprint in a peer reviewed journal and replaced system, while also providing the best possible care the original assessment. No restrictions were made for patients, reliable prognosis of covid-19 remains on the setting (eg, inpatients, outpatients, or general important to inform decisions regarding shielding, population), prediction horizon (how far ahead the vaccination, treatment, and hospital or intensive care model predicts), included predictors, or outcomes. unit (ICU) admission. Prediction models that combine Epidemiological studies that aimed to model disease several variables or features to estimate the risk of transmission or fatality rates, and predictor finding people being infected or experiencing a poor outcome studies, were excluded. We only included studies from the infection could assist medical staff in triaging published in English. Starting with the second update, patients when allocating limited healthcare resources. retrieved records were initially screened by a text 2 doi: 10.1136/bmj.m1328 | BMJ 2020;369:m1328 | the bmj  Research analysis tool developed using artificial intelligence to measures included any summaries of discrimination prioritise sensitivity (supplementary material). Titles, (the extent to which predicted risks discriminate abstracts, and full texts were screened for eligibility in between participants with and without the outcome), duplicate by independent reviewers (pairs from LW, and calibration (the extent to which predicted risks BVC, MvS, and KGMM) using EPPI-Reviewer,11 and correspond to observed risks) as recommended in BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. discrepancies were resolved through discussion. the TRIPOD (transparent reporting of a multivariable Data extraction of included articles was done by prediction model for individual prognosis or two independent reviewers (from LW, BVC, GSC, TPAD, diagnosis; https://www.tripod-statement.org/) Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. MCH, GH, KGMM, RDR, ES, LJMS, EWS, KIES, CW, AL, statement.15 16 Discrimination is often quantified by JM, TT, JAD, KL, JBR, LH, CS, MS, MCH, NS, NK, SMJvK, the C index (C index=1 if the model discriminates JCS, PD, CLAN, RW, GPM, IT, JYV, DLD, JW, FSvR, PH, perfectly; C index=0.5 if discrimination is no better VMTdJ, BCTvB, ICCvdH, DJM, MK, BL, EA, SG, BA, than chance). Calibration is often assessed graphically JH, KJ, SG, KR, JE, MH, VB, and MvS). Reviewers used using calibration plots or quantified by the calibration a standardised data extraction form based on the intercept (which is zero when the risks are not CHARMS (critical appraisal and data extraction for systematically overestimated or underestimated) and systematic reviews of prediction modelling studies) calibration slope (which is one if the predicted risks checklist12 and PROBAST (prediction model risk of are not too extreme or too moderate).17 We focused on bias assessment tool; https://www.probast.org/) for performance statistics as estimated from the strongest assessing the reported prediction models.13 14 We available form of validation (in order of strength: sought to extract each model’s predictive performance external (evaluation in an independent database), by using whatever measures were presented. These internal (bootstrap validation, cross validation, 126 969 9 Records identified through database searching Additional records identified through other sources 126 978 Records screened 126 150 Records excluded 828 Articles assessed for eligibility 416 Articles excluded 120 Not a prediction model development or validation study 84 Preprint released aer 5 May 2020 89 Diagnostic model published aer 17 February 2021 30 Epidemiological model to estimate disease transmission or case fatality rate 40 Methods paper 21 Commentary, editorial, or letter 21 Duplicate article 5 No full text 5 Written in foreign language (eg, Chinese) 1 Prognostic model based images alone published aer 17 February 2021 412 Studies included in review (731 models) 125 13 593 Diagnostic models (including Models identifying people Prognostic models (including 12 severity models and at risk in general population 265 for mortality, 84 for 75 imaging studies) progression to severe or critical state) Included in supplementary material and project website Included in present analysis (310 studies with 606 models) Fig 1 | PRISMA (preferred reporting items for systematic reviews and meta-analyses) flowchart of study inclusions and exclusions the bmj | BMJ 2020;369:m1328 | doi: 10.1136/bmj.m1328 3 Research table 1).19-430 Of these, 310 studies describing 606 Box 1: Availability of models in format for use in clinical practice prognostic models or validations of prognostic models Three hundred and eighty one unique prognostic models were developed in the are included in the current analysis: 13 prognostic included studies. Eighty (21%) of these models were presented as a model equation models for developing covid-19 in the general including intercept and regression coefficients. Thirty nine (10%) models were only population and 593 prognostic models for predicting BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. partially presented (eg, intercept or baseline hazard were missing). The remaining 262 outcomes in patients with covid-19 diagnoses. The (69%) did not provide the underlying model equation. results from previous updates, including diagnostic One hundred and sixty one (42%) were available in a tool to facilitate use in clinical models, are available as supplementary material. practice (in addition to or instead of a published equation). Sixty models (16%) were Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. A database with the description of each model and presented as a nomogram, 35 (9%) as a web calculator, 30 (8%) as a sum score, validation and its risk of bias assessment can be found nine (2%) as a software object, five (1%) as a decision tree or set of predictions for on https://www.covprecise.org/. subgroups, and 22 (6%) in other usable formats (6%). Of the 606 prognostic models, 381 were unique, All these presentation formats make predictions readily available for use in the newly developed models for covid-19, and 225 were clinic. However, because most of these prognostic models were at high or uncertain external validations of existing models in a study other risk of bias, we do not recommend their routine use before they are externally than the model development study. These external validated, ideally by independent investigators in other data than used for their validations include external validations of newly development. developed covid models, as well as prognostic scores predating the covid-19 pandemic. Some models were validated more than once (in different studies, as random training test splits, temporal splits), apparent described below). One hundred and fifty eight (41%) (evaluation by using exactly the same data used for newly developed models were publicly available in a development)). Any discrepancies in data extraction format for use in practice (box 1). were discussed between reviewers, and remaining conflicts were resolved by LW or MvS. The online Primary datasets supplementary material provides details on data Five hundred and fifty six (92%) developed or validated extraction. Some studies investigated multiple models models used data from a single country (table 1), 39 and some models were investigated in multiple studies (6%) used international data, and for 11 (2%) models (that is, in external validation studies). The unit of it was unclear how many (and which) countries analysis was a model within a study, unless stated contributed data. Three (0.5%) models used simulated otherwise. We considered aspects of PRISMA (preferred data and 21 (3%) used proxy data to estimate covid-19 reporting items for systematic reviews and meta- related risks (eg, Medicare claims data from 2015 to analyses)18 and TRIPOD15 16 in reporting our study. 2016). Most models were intended for use in confirmed Details on all reviewed studies and prediction models covid-19 cases (83%) and a hospital setting (82%). are publicly available at https://www.covprecise.org/. The average patient age ranged from 38 to 84 years, and the proportion of men ranged from 1% to 95%, Patient and public involvement although this information was often not reported. Severe covid-19 survivors and lay people participated Based on the studies that reported study dates, data by discussing their perspectives, providing advice, were collected from December 2019 to October 2020. and acting as partners in writing a lay summary of Some centres provided data to multiple studies and the project’s aims and results (available at https:// it was unclear how much these datasets overlapped www.covprecise.org/project/), thereby taking part in across identified studies. the implementation of knowledge distribution. Owing The median sample size for model development to the initial emergency situation of the covid-19 was 414, with a median number of 74 individuals pandemic, we did not involve patients or the public in experiencing the event that was predicted. The the design and conduct of this living review in March mortality risk in patients admitted to hospital ranged 2020, but the study protocol and preliminary results from 8% to 46%. This wide variation is partly due to were immediately publicly available on https://osf.io/ differences in length of follow-up between studies ehc47/, medRxiv, and https://www.covprecise.org/ (which was often not reported), local and temporal living-review/. variation in diagnostic criteria, admission criteria and treatment, as well as selection bias (eg, excluding Results participants who had neither recovered nor died at the We identified 126 969 records through our systematic end of the study period). search, of which 89 566 were identified in the present search update (supplementary table 1, fig 1). We Models to predict covid-19 related risks in the included a further nine studies that were publicly general population available but were not detected by our search. Of We identified 13 newly developed models aiming to 126 978 titles, 828 studies were retained for abstract predict covid-19 related risks in the general population. and full text screening. We included 412 studies Five models predicted hospital admission for covid-19, describing 731 prediction models or validations, of three predicted mortality, one predicted development which 243 studies with 499 models or validations were of severe covid-19, and four predicted an insufficiently newly included in the present update (supplementary defined covid-19 outcome. Eight of these 13 general 4 doi: 10.1136/bmj.m1328 | BMJ 2020;369:m1328 | the bmj  Research Table 1 | Characteristics of reviewed prediction models for prognosis of coronavirus 84%), but a specific intended use (ie, when exactly or disease 2019 (covid-19) at which moment in the investigation to use them, and No (%) of models* or median for whom) was often not clearly described. Of these (interquartile range) 593 prognostic models, 265 (45%) estimated mortality Country† risk, 84 (14%) predicted progression to a severe or BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. Single country data 556 (92) critical disease, and 53 (9%) predicted ICU admission. China 200 (33) US 80 (13) The remaining 191 studies used other outcomes Italy 59 (10) (single or as part of a composite) including need for Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. UK 53 (9) intubation, (duration of) mechanical ventilation, Spain 40 (7) oxygen support, acute respiratory distress syndrome, South Korea 28 (5) septic shock, cardiovascular complications, (multiple) Mexico 21 (3) organ failure, and thrombotic complication, length Turkey 15 (2) Norway 10 (2) of hospital stay, recovery, hospital admission or Other single country 50 (8) readmission, or length of isolation period. Prediction International (combined) data 39 (6) horizons varied between one day and 60 days but were Unknown origin of data 11 (2) often unspecified (n=387, 65%). Some studies (n=13, Type of data used 2%) used proxy outcomes. For example, one study Pandemic data 582 (96) used data from 2015 to 2019 to predict mortality and Proxy (non-covid-19) data 21 (3) Simulated data 3 (0.5) prolonged assisted mechanical ventilation (as a non- Target setting covid-19 proxy outcome).119 Patients admitted to hospital 496 (82) The studies reported C indexes between 0.49 and Patient at triage centre or fever clinic 6 (1) 1, with a median of 0.81 (interquartile range 0.75- Patients in general practice 3 (0.5) 0.89). The median C index was 0.83 for the mortality Other 54 (9) models, 0.83 for progression models, and 0.77 for ICU Unclear 47 (8) admission models. Researchers showed calibration Target population Confirmed covid-19 502 (83) plots for only 152 of the 593 models (26%, of which Suspected covid-19 46 (8) 102 at external validation). The calibration results Other 25 (4) were mixed, with several studies indicating inaccurate Unclear 33 (5) risk predictions (examples in Xie et al,19 Barda et al,73 Type of model and Zhang et al122). Plots were sometimes constructed Prognostic model to predict future risk of covid-19 in general 13 (2) population in an unclear way, hampering interpretation (examples Prognostic models for outcomes in patients with covid-19 593 (98) in Guo et al,89 Gong et al,125 and Knight et al147). Analysis done in reviewed study Development only 96 (16) Risk of bias Development and internal validation 185 (31) Seven newly developed prognostic models and 22 Development and external validation 100 (17) external validations of prognostic models were at low External validation only 225 (37) Sample size risk of bias (n=29, 5%). Most newly developed models Sample size (development) 414 (172-1505) and external validations were at unclear (n=32, 5%) or No of events (development) 74 (36-207) high (n=545, 90%) risk of bias according to assessment Sample size (external validation) 314 (127-516) with PROBAST, which suggests that the predictive No of events (external validation) 42 (24-115) performance when used in practice is probably lower *Analysis unit is a model within a study. Some studies investigated multiple models and some models were investigated in multiple studies (that is, in external validation studies). than what is reported (fig 2). Figure 2 and box 2 give †A study that uses development data from one country and validation data from another is classified as details on common causes for risk of bias. international. Three hundred and eighty four (63%) of the 606 models and validations had a low risk of bias for the participants domain. One hundred and seven models population models used proxy outcomes (eg, (18%) had a high risk of bias for the participants admission for non-tuberculosis pneumonia, influenza, domain, which indicates that the participants acute bronchitis, or upper respiratory tract infections enrolled in the studies might not be representative of instead of hospital admission for covid-19).20 The 13 the models’ targeted populations. Unclear reporting studies reported C indexes between 0.52 and 0.99. on the inclusion of participants led to an unclear Calibration was assessed for only four models, all in risk of bias assessment in 115 models (19%). Three one study, which found slight miscalibration.231 hundred and eighty six models (64%) had a low risk of bias for the predictor domain, while 193 (32%) Prognostic models for outcomes in patients with had an unclear risk of bias and 27 had a high risk of diagnosis of covid-19 bias (4%). High risk of bias for the predictor domain We identified 593 prognostic models for predicting indicates that predictors were not available at the clinical outcomes in patients with covid-19 (368 models’ intended time of use, not clearly defined, or developments, 225 external validations). These influenced by the outcome measurement. Most studies models were primarily for use in patients admitted to used outcomes that are easy to assess (eg, all cause hospital with a proven diagnosis of covid-19 (n=496, death), and hence 353 (58%) were rated at low risk of the bmj | BMJ 2020;369:m1328 | doi: 10.1136/bmj.m1328 5 Research Risk of bias All (n=606) High Unclear Low 100 Percentage of models 75 BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. 50 Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. 25 0 General population (n=13) 100 Percentage of models 75 50 25 0 Prognosis (n=593) 100 Percentage of models 75 50 25 0 Overall Participants Predictors Outcome Analysis PROBAST domain Fig 2 | PROBAST (prediction model risk of bias assessment tool) risk of bias for all included models combined (n=606) and broken down per type of analysis bias for the outcome domain. Nonetheless, there was The four Qcovid models predict hospital admission cause for concern about bias induced by the outcome and death with covid-19 in the general population measurement in 106 models (17%), for example, due in the UK, separately for men and women.231 The to the use of proxy outcomes (eg, hospital admission models use age, ethnic group, deprivation, body mass for non-covid-19 severe respiratory infections). One index, and a range of comorbidities as predictors. The hundred and forty seven (24%) had an unclear risk of models showed underestimated risks for high risk bias due to opaque or ambiguous reporting. In contrast patients at external validation, which was remedied by to the participant, predictor, and outcome domains, recalibrating the model.231 the analysis domain was problematic for most of the The PRIEST score262 predicts 30 day death or organ 606 models and validations. Overall, 530 (87%) were support in patients with suspected or confirmed at high risk of bias for the analysis domain, and the covid-19 presenting at the emergency department. reporting was insufficiently clear to assess risk of bias The triage score is based on NEWS2 (national early in the analysis in 42 (7%). Only 34 (6%) were at low warning score 2 consisting of respiratory rate, oxygen risk of bias for the analysis domain. saturation, heart rate, systolic blood pressure, temperature, consciousness, air, or supplemental Newly developed models at low risk of bias oxygen), age, sex, and performance status (ranging We found seven newly developed models at low risk of from bed-bound to normal performance). Its external bias (table 2). All had good to excellent discrimination, validation in UK emergency departments showed but calibration varied, highlighting the need of local reasonable calibration, but potential heterogeneity in and temporal recalibration. calibration across centres was not examined. 6 doi: 10.1136/bmj.m1328 | BMJ 2020;369:m1328 | the bmj  Research Box 2: Common causes of risk of bias in the reported prediction models of covid-19 The analysis domain was the most problematic domain: 87% (n=530) of newly developed models and validations were at high risk of bias, compared to 18% (n=107), 4% (n=27), and 17% (n=106) for the participant, predictor, and outcome domains. One hundred and fifty one (25%) models had low risk of bias on all domains except analysis, indicating adequate data collection and study design, but issues that could have been avoided by BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. conducting a better statistical analysis. The most frequent problem was insufficient sample size (n=408, 67%). Small to modest sample sizes and numbers of events (table 1) led to an increased risk of overfitting, particularly if complex modelling strategies were used. Not properly accounting for overfitting or optimism was also common (n=250, 41%). Ninety six models (16%) were neither internally nor externally validated. If done, internal validation was sometimes not correctly executed (ie, not all modelling steps were repeated). Performance statistics from these models are Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. likely optimistic. Moreover, evaluation of discrimination and calibration was often incomplete, or done with inappropriate statistics (n=338, 56%). Calibration was only assessed for 156 models using calibration plots (26%), of which 106 (17%) on external validation data. Inappropriate handling of missing data was common (n=290, 48%). One hundred and twenty seven conducted a complete case analysis (21%), 205 (34%) did not mention how missing data was handled. Models to predict covid-19 risk in general population versus prognostic models in patients with covid-19 The 593 prognostic models for patients with covid-19 were more often at high risk of bias than the 13 general population models (90% (n=536) v 69% (n=9)). This difference was mainly due to the analysis domain (88% (n=521) v 69% (n=9) at high risk of bias). The median sample size for model development in patients with covid-19 was 397 (71 events), compared to >1.6 million (1867 events) for general population models. The median sample size for external validation was 299 (42 events), compared to >1 million (1303 events) for general population models. Hence, more models had an inadequate sample size for the chosen analysis strategy (69% (n=407) v 8% (n=1)), and more were at risk of overfitting and optimism (42% (n=248 v 15% (n=2)). The outcome domain was more problematic for the general population models than for the models for patients with covid-19, with 62% (n=8) versus 17% (n=98) at high risk of bias in this domain. This difference was caused using proxy outcomes (n=8, 62%)—for example, hospital admission due to severe respiratory disease other than covid-19. For the participant and predictor domains, the risk of bias was comparable (fig 2). Development and external validation External validations were more often at low risk of bias than newly developed models (10%, (n=22/225) v 2% (n=7/381)). The statistical analysis domain was the most problematic domain for model development as well as for external validation studies, with 93% (n=353) and 79% (n=177) at high risk of bias for this domain, respectively. The most common causes of high risk of bias were the same for both types (small sample size, inappropriate evaluation of predictive performance, and inappropriate handling of missing data), except for overfitting and optimism, which is not a concern at external validation. Carr’s model81 and the ISARIC 4C deterioration study using single-centre UK data (table 3).269 This model268 predict deterioration in covid-19 patients validation study included 411 patients, of which 180 admitted to hospital. The composite outcomes for experienced a deterioration in health, and 115 died. In both models included ICU admission and death, while this study, the Carr model and NEWS2 performed best the ISARIC 4C model also adds ventilatory support. to predict deterioration, while the Xie model and REMS Both models had comparable performance but performed best to predict mortality. Both the Carr included different predictors, all typically available model (a preprint version that differs slightly from the at admission. Carr and colleagues supplemented Carr model reported above) and the Xie model showed NEWS2 with age, laboratory and physiological slight miscalibration. parameters (supplemental oxygen flow rate, urea, NEWS2 and REMS were also validated in other oxygen saturation, C reactive protein, estimated dedicated validation studies. NEWS2 obtained C indexes glomerular filtration rate, neutrophil count, between 0.65 and 0.90.141 203 214 233 245 280 281 303 319 340 neutrophil-lymphocyte ratio). Gupta and colleagues268 REMS obtained C indexes between 0.74 and 0.88.91 233 319 developed a model including age, sex, nosocomial These studies were too heterogeneous and biased to infection, Glasgow coma scale score, peripheral meta-analyse: they used varying outcome definitions oxygen saturation at admission, breathing room air or (mortality, ICU admission, various composites, with oxygen therapy, respiratory rate, urea concentration, C time horizons varying from 1 to 30 days), from different reactive protein concentration, lymphocyte count, and populations (Italy, UK, Norway, China), and were at high presence of radiographic chest infiltrates. Carr’s model or unclear risk of bias. was validated internationally81 and the ISARIC4C Deterioration model was validated regionally within Discussion the UK.268 For both models, calibration varied across In this systematic review of prognostic prediction settings. models related to the covid-19 pandemic, we identified and critically appraised 606 models described in 310 External validations at low risk of bias studies. These prognostic models can be divided into We identified 225 external validations in dedicated models to predict the risk of developing covid-19 (ie, not combined with the development of the model) or having an adverse disease course in the general external validation studies. Only 22 were low risk of population (n=13), and models to support the bias, although all 22 validations came from the same prognosis of patients with covid-19 (n=593). Most the bmj | BMJ 2020;369:m1328 | doi: 10.1136/bmj.m1328 7 Research Table 2 | Prediction models for covid-19 with low risk of bias Predictive performance Sample size (total No of partici- Strongest type of Study; setting; and outcome Model pants (No with outcome)) validation reported C index (95% CI)* General population models BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. Clift et al231; data from UK, men registered at GP; death with Qcovid mortality (male) Development 3 047 693 (1867); External validation, new 0.93 (0.92 to 0.94) covid-19 external validation 1 097 268 (744) centres, same country Clift et al231; data from UK, women registered at GP; death Qcovid mortality (female) Development 3 035 409 (2517); External validation, new 0.93 (0.92 to 0.94) with covid-19 external validation 1 075 788 (978) centres, same country Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. Clift et al231; data from UK, men registered at GP; hospital Qcovid hospital Development 3 047 693 (5962); External validation, new 0.86 (0.85 to 0.87) admission for covid-19 or death with covid-19 admission (male) external validation 1 097 268 centres, same country (2076) Clift et al231; data from UK, women registered at GP; hospital Qcovid hospital Development 3 035 409 (4814); External validation, new 0.85 (0.84 to 0.86) admission for covid-19 or death with covid-19 admission (female) external validation 1 075 788 (1627) centres, same country Models for patients with covid-19 Carr et al81; data from UK, China and Norway; patients Carr model Development 1 276 (389); external External validation, 0.79 (not reported) admitted to hospital with confirmed covid-19; 14 day ICU validation 6237 (1308) new centres, different admission or death countries Goodacre et al262; data from UK; patients with suspected PRIEST score Development 11 773 (2440); External validation, new 0.80 (0.79 to 0.81) symptoms of covid-19 at the emergency department; 30 day external validation 9118 centres, same country death or organ support Gupta et al268; data from the UK; hospitalised symptomatic ISARIC4C Deterioration Development 66 705 (28 140); External validation, new 0·77 (0·76 to 0·78) suspected or confirmed cases; ventilatory support, critical model external validation 8 239 (3 784) centers, same country care, or in-hospital death GP=general practice; ICU=intensive care unit; PRIEST=Pandemic Respiratory Infection Emergency System Triage; ISARIC4C=International Severe Acute Respiratory and Emerging Infections Consortium Coronavirus Clinical Characterisation Consortium; CI= confidence interval. *Performance from strongest type of validation reported. studies reported moderate to excellent predictive or external validation (Xie’s model19 269). We suggest performance, but only seven newly developed models that these models should be further validated within and 22 external validations of existing models were at other datasets and settings, and ideally by independent low risk of bias. From these, we identified eight models, investigators, to investigate which models maintain a all developed for prognosis of covid-19, with adequate robust performance over time and in varying settings. performance and low risk of bias at model development Most of the 606 models were appraised to have high (four Qcovid models,231 the PRIEST model,262 the or uncertain risk of bias owing to a combination of poor ISARIC4C deterioration model,268 and Carr’s model81) reporting and poor methodological conduct. Often, the available sample sizes and number of events for the Table 3 | External validations with low risk of bias from Gupta et al269 outcomes of interest were limited. This problem is well Outcome Model C index (95% CI) known when building prediction models and increases 1 day deterioration NEWS2431 0.78 (0.73 to 0.83) the risk of overfitting the model.438 Other common 10 day deterioration Ji94 0.56 (0.50 to 0.62) causes for bias were not adequately accounting for 14 day deterioration Carr (pre-print*)432 0.78 (0.74 to 0.82) missing data, using techniques that do not account for 14 day deterioration Carr (preprint threshold*) 0.76 (0.71 to 0.81) optimism in performance estimates, ignoring model 14 day deterioration Guo89 0.67 (0.61 to 0.73) calibration, and inappropriate model validation. In-hospital deterioration Zhang (poor†)122 0.74 (0.69 to 0.79) A high risk of bias implies that the performance of In-hospital deterioration Galloway139 0.72 (0.68 to 0.77) In-hospital deterioration TACTIC433 0.70 (0.65 to 0.75) these models in new samples will probably be worse In-hospital deterioration Colombi85 0.69 (0.63 to 0.74) than that reported by the researchers. Therefore, the In-hospital deterioration Huang66 0.67 (0.1 to 0.73) estimated C indexes, often indicating near perfect In-hospital deterioration Shi43 0.61 (0.56 to 0.66) discrimination, are probably optimistic. For most of In-hospital deterioration MEWS434 0.60 (0.54 to 0.65) these models, no independent external validations 12 day mortality Lu26 0.72 (0.67 to 0.76) with a low risk of bias were performed, even though 30 day mortality CURB-65435 0.75 (0.70 to 0.80) 30 day mortality Bello-Chavolla76 0.66 (0.60 to 0.72) most were publicly available in a format usable in In-hospital mortality REMS436 0.76 (0.71 to 0.81) clinical practice. In-hospital mortality Xie19 0.76 (0.69 to 0.82) In-hospital mortality Hu91 0.74 (0.68 to 0.79) Challenges and opportunities In-hospital mortality Caramelo25 0.71 (0.66 to 0.76) The main aim of prediction models is to support In-hospital mortality Zhang (death†)122 0.70 (0.65 to 0.76) medical decision making in individual patients. In-hospital mortality qSOFA437 0.60 (0.55 to 0.65) Therefore, it is vital to identify a target setting in In-hospital mortality Yan28 0.58 (0.49 to 0.67) NEWS2=national early warning score 2; TACTIC=therapeutic study in pre-ICU patients admitted with covid-19; which predictions serve a clinical need (eg, emergency MEWS=modified early warning score; REMS=rapid emergency medicine score; qSOFA=quick sequential (sepsis- department, intensive care unit, general practice, related) organ failure assessment; CURB-65=confusion, urea, respiratory rate, blood pressure plus age of at least 65 years; CI=confidence interval. symptom monitoring app in the general population), *Preprint of the study by Carr et al432 contains a model with and without a threshold. Both were validated and a representative dataset from that setting separately by Gupta et al. (preferably comprising consecutive patients) on which †Preprint of the study by Zhang et al122 contains a model for poor outcomes (defined originally as developing ARDS, need for intubation or extracorporeal membrane oxygenation support, ICU admission and death), and a the prediction model can be developed and validated. model for death. Both were validated separately. This clinical setting and patient characteristics 8 doi: 10.1136/bmj.m1328 | BMJ 2020;369:m1328 | the bmj  Research should be described in detail (including timing A prediction model applied in a new healthcare within the disease course, the severity of disease at setting or country often produces predictions that are the moment of prediction, and the comorbidity), so miscalibrated447 and might need to be updated before it that readers and clinicians are able to understand can safely be applied in that new setting.17 This requires if the proposed model is suited for their population. data from patients with covid-19 to be available from BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. However, the studies included in our systematic that setting. In addition to updating predictions in review often lacked an adequate description of the their local setting, individual participant data from target setting and study population, which leaves multiple countries and healthcare systems might Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. users of these models in doubt about the models’ allow better understanding of the generalisability and applicability. Although we recognise that the earlier implementation of prediction models across different studies were done under severe time constraints, we settings and populations. This approach could greatly recommend that researchers adhere to the TRIPOD improve the applicability and robustness of prediction reporting guideline15 16 to improve the description models in routine care.446 449-452 of their study population and guide their modelling The covid-19 pandemic has been characterised by an choices. TRIPOD translations (eg, in Chinese) are unprecedented speed of data accumulation worldwide. also available at https://www.tripod-statement.org. A Unfortunately, much of the work done to analyse all better description of the study population could also these data has been ill informed and disjointed. As a help us understand the observed variability in the result, we have hundreds of similar models, and very reported outcomes across studies, such as covid-19 few independent validation studies comparing their related mortality. The variability in mortality could performance on the same data. To leverage the full be related to differences in included patients (eg, age, potential of prediction models in emerging pandemics comorbidities) but also in interventions for covid-19. and quickly identify useful models, international In this living review, inadequate sample size to and interdisciplinary collaboration in terms of data build a robust model or to obtain reliable performance acquisition, model building, model validation, and statistics was one of the most prevalent shortcomings. systematic review is crucial. We recommend researchers should make use of formulas and software that have been made available Study limitations in recent years to calculate the required sample size With new publications on covid-19 related to build or externally validate models.439-442 The prediction models entering the medical literature in current review also identified that ignoring missing unprecedented numbers and at unprecedented speed, data and performing a complete case analysis is still this systematic review cannot be viewed as an up- very common. As this leads to reduced precision to-date list of all currently available covid-19 related and can introduce bias in the estimated model, we prediction models. It does provide a comprehensive recommend researchers address missing data using overview of all prognostic model developments and appropriate techniques before developing or validating validations from the first year of the pandemic up to a model.443 444 When creating a new prediction model, 17 February 2021. Also, 69 of the studies we reviewed we recommend building on previous literature and were only available as preprints. Some of these expert opinion to select predictors, rather than selecting studies might enter the official medical literature in predictors purely based on data.17 This recommendation an improved version, after peer review. We reassessed is especially important for datasets with limited sample peer reviewed publications of preprints included in size.445 To temper optimism in estimated performance, previous updates that have been published before several internal validation strategies can be used—for the current update. We also found other prediction example, bootstrapping.17 446 We also recommend models have been used in clinical practice without that researchers should evaluate model performance scientific publications,453 and web risk calculators in terms of correspondence between predicted and launched for use while the scientific manuscript is still observed risk, preferably using flexible calibration under review (and unavailable on request).454 These plots17 447 in addition to discrimination. unpublished models naturally fall outside the scope Covid-19 prediction will often not present as a of this review of the literature. As we have argued simple binary classification task. Complexities in the extensively elsewhere,455 transparent reporting that data should be handled appropriately. For example, a enables validation by independent researchers is key prediction horizon should be specified for prognostic for predictive analytics, and clinical guidelines should outcomes (eg, 30 day mortality). If study participants only recommend publicly available and verifiable neither recovered nor died within that period, their algorithms. data should not be excluded from analysis, which some reviewed studies have done. Instead, an appropriate Implications for practice time-to-event analysis should be considered to allow This living review has identified a handful of models for administrative censoring.17 Censoring for other developed specifically for covid-19 prognosis with reasons, for instance because of quick recovery and good predictive performance at external validation, loss to follow-up of patients who are no longer at risk and with model development or external validation of death from covid-19, could necessitate analysis in a at low risk of bias. The Qcovid models231 were built competing risk framework.448 to prognosticate hospital admission and mortality the bmj | BMJ 2020;369:m1328 | doi: 10.1136/bmj.m1328 9 Research risks in the general population. The PRIEST model clinicians and policymakers to prematurely implement was proposed to triage patients at the emergency prediction models without sufficient documentation department.262 The ISARIC4C Deterioration model,268 and validation. Inaccurate models could even cause Carr model,81 and Xie model19 269 were developed to more harm than good.461 By pointing to the most predict adverse outcomes in hospitalised patients important methodological challenges and issues in BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. (ventilatory support, critical care or death, ICU design and reporting, we hope to have provided a admission or death, and death, respectively). Since useful starting point for future studies and future the search date, these models have been validated epidemics. Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. temporally and geographically, which demonstrated that care should be taken when using these models Conclusion in policy or clinical practice.231 268 456-460 Differences Several prognostic models for covid-19 are currently between healthcare systems, fluctuations in infection available and most report moderate to excellent rates, virus mutations, differences in vaccination discrimination. However, many of these models are status, varying testing criteria, and changes in patient at high or unclear risk of bias, mainly because of management and treatment can lead to miscalibration model overfitting, inappropriate model evaluation (eg, in more recent or local data. Hence, future studies calibration ignored), and inappropriate handling of should focus on validating and comparing these missing data. Therefore, their performance estimates prediction models with low risk of bias.17 External are probably optimistic and might not be representative validations should not only assess discrimination, for the target population. We found that the Qcovid but also calibration and clinical usefulness (net models can be used for risk stratification in the general benefit),447 452 461 in large studies439 440 442 462 463 using population, while the PRIEST model, ISARIC4C an appropriate design. Deterioration model, Carr’s model, and Xie’s model Many prognostic models have been developed are suitable for prognostication in a hospital setting. for prognostication in a hospital setting. Updating The performance of these models is likely to vary an available model to accommodate temporal or over time and differ between regions, necessitating regional differences or extending an existing model further validation and potentially updating before with new predictors requires less data and provides implementation. For details of the reviewed models, generally more robust predictions than developing see https://www.covprecise.org/. Sharing data and a new prognostic model.17 New variants could vary expertise for the validation and updating of covid-19 in contagiousness and severity, and vaccination related prediction models is still needed. and waning immunity might alter individual risks. Consequently, even updated models could become Author affiliations 1 Department of Epidemiology, CAPHRI Care and Public Health outdated. These changes would primarily affect Research Institute, Maastricht University, Maastricht, Netherlands calibration (ie, absolute risk estimates might be too 2 Department of Development and Regeneration, KU Leuven, high or too low), while the discrimination between Leuven, Belgium low and high risk patients could be less affected. 3 Department of Biomedical Data Sciences, Leiden University Miscalibration is especially concerning for general Medical Centre, Leiden, Netherlands 4 population models. Models that focus on patients Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Musculoskeletal Sciences, University of Oxford, seeking care and adjust risk estimates for symptoms Oxford, UK and severity markers might be more robust, but this 5 NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, hypothesis remains to be confirmed empirically. Oxford, UK Although many models exist to predict outcomes at 6 Centre for Prognosis Research, School of Medicine, Keele the emergency department or at hospital admission, University, Keele, UK 7 few are suited for patients with symptoms attending Section for Clinical Biometrics, Centre for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, primary care, or for patients admitted to the ICU. Vienna, Austria In addition, the models reviewed so far focus on the 8 Julius Center for Health Sciences and Primary Care, University covid-19 diagnosis or assess the risk of mortality Medical Center Utrecht, Utrecht University, Utrecht, Netherlands 9 or deterioration, whereas long term morbidity and Cochrane Netherlands, University Medical Center Utrecht, Utrecht functional outcomes remain understudied and could University, Utrecht, Netherlands 10 Department of Hygiene and Epidemiology, University of Ioannina be a target outcome of interest in future studies Medical School, Ioannina, Greece developing prediction models.464 465 11 Department of Medical Microbiology, University Medical Centre This review of prediction models developed in the Utrecht, Utrecht, Netherlands first year of the covid-19 pandemic found most models 12 HRB Clinical Research Facility, Cork, Ireland at unclear or high risk of bias. Whereas many external 13 School of Public Health, University College Cork, Cork, Ireland validations were done, most were at high risk of bias 14 Smart Data Analysis and Statistics BV, Utrecht, Netherlands and most models developed specifically for covid-19 15 Department of Electrical Engineering, ESAT Stadius, KU Leuven, were not validated independently. This oversupply of Leuven, Belgium 16 insufficiently validated models is not useful for clinical Ordensklinikum Linz, Hospital Elisabethinen, Department of Nephrology, Linz, Austria practice. Moreover, the urgency of diagnostic and 17 Department of Biostatistics, Epidemiology and Informatics, prognostic models to assist in quick and efficient triage Perelman School of Medicine, University of Pennsylvania, of patients in an emergent pandemic might encourage Philadelphia, PA, USA 10 doi: 10.1136/bmj.m1328 | BMJ 2020;369:m1328 | the bmj  Research 18 Palliative and Advanced Illness Research Center and Division of of the data analysis, and had final responsibility for the decision to Pulmonary and Critical Care Medicine, Department of Medicine, submit for publication. The corresponding author attests that all listed Perelman School of Medicine, University of Pennsylvania, authors meet authorship criteria and that no others meeting the Philadelphia, PA, USA criteria have been omitted. 19 Department of Microbiology, Immunology and Transplantation, KU Funding: LW, BVC, LH, and MDV acknowledge specific funding Leuven-University of Leuven, Leuven, Belgium for this work from Internal Funds KU Leuven, KOOR, and the BMJ: first published as 10.1136/bmj.m1328 on 7 April 2020. Downloaded from https://www.bmj.com/ on 6 February 2025 by guest. 20 Department of General Internal Medicine, KU Leuven-University covid-19 Fund. LW is a postdoctoral fellow of Research Foundation- Hospitals Leuven, Leuven, Belgium Flanders (FWO) and receives support from ZonMw (grant 21 10430012010001). BVC received support from FWO (grant Population Health Research Institute, St. George’s University of G0B4716N) and Internal Funds KU Leuven (grant C24/15/037). London, Cranmer Terrace, London, UK TPAD acknowledges financial support from the Netherlands Protected by copyright, including for uses related to text and data mining, AI training, and similar technologies. 22 Department of Nephrology, Medical University of Vienna, Vienna, Organisation for Health Research and Development (grant Austria 91617050). VMTdJ was supported by the European Union Horizon 23 Evidence-Based Oncology, Department I of Internal Medicine and 2020 Research and Innovation Programme under ReCoDID grant Centre for Integrated Oncology Aachen Bonn Cologne Dusseldorf, agreement 825746. KGMM and JAD acknowledge financial support Faculty of Medicine and University Hospital Cologne, University of from Cochrane Collaboration (SMF 2018). KIES is funded by the Cologne, Cologne, Germany National Institute for Health Research (NIHR) School for Primary 24 Care Research. The views expressed are those of the author(s) and Department of Clinical Epidemiology, Leiden University Medical not necessarily those of the NHS, the NIHR, or the Department of Centre, Leiden, Netherlands Health and Social Care. GSC was supported by the NIHR Biomedical 25 Division of Informatics, Imaging and Data Science, Faculty of Research Centre, Oxford, and Cancer Research UK (programme grant Biology, Medicine and Health, Manchester Academic Health Science C49297/A27294). JM was supported by the Cancer Research UK Centre, University of Manchester, Manchester, UK (programme grant C49297/A27294). PD was supported by the 26 Institute of Applied Health Sciences, University of Aberdeen, NIHR Biomedical Research Centre, Oxford. MOH is supported by Aberdeen, UK the National Heart, Lung, and Blood Institute of the United States 27 National Institutes of Health (grant R00 HL141678). ICCvDH and Centre for Biostatistics, University of Manchester, Manchester BCTvB received funding from Euregio Meuse-Rhine (grant Covid Data Academic Health Science Centre, Manchester, UK Platform (coDaP) interreg EMR-187). BL was supported by a Fonds 28 Centre for Epidemiology Versus Arthritis, Centre for de recherche du Québec-Santé postdoctoral training fellowship. JYV Musculoskeletal Research, University of Manchester, Manchester acknowledges the National Institute for Health and Care Research Academic Health Science Centre, Manchester, UK (NIHR) Community Healthcare MedTech and In Vitro Diagnostics 29 Division of Nursing, Midwifery and Social Work, School of Health Co-operative at Oxford Health NHS Foundation Trust. The funders Sciences, University of Manchester, Manchester, UK had no role in study design, data collection, data analysis, data 30 interpretation, or reporting. Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK Competing interests: All authors have completed the ICMJE uniform 31 disclosure form at www.icmje.org/disclosure-of-interest/ and declare: Amsterdam UMC, University of Amsterdam, Amsterdam Public support from Internal Funds KU Leuven, KOOR, and the covid-19 Fund Health, Medical Library, Netherlands for the submitted work; no competing interests with regards to the 32 Department of General Medicine, Shirakawa Satellite for Teaching submitted work; LW discloses support from Research Foundation- And Research, Fukushima Medical University, Fukushima, Japan Flanders; RDR reports personal fees as a statistics editor for The BMJ 33 Department of Epidemiology and Biostatistics, Imperial College (since 2009), consultancy fees for Roche for giving meta-analysis London School of Public Health, London, UK teaching and advice in October 2018, and personal fees for delivering 34 in-house training courses at Barts and the London School of Medicine Department of Clinical Epidemiology and Medical Technology and Dentistry, and the Universities of Aberdeen, Exeter, and Leeds, Assessment, Maastricht University Medical Centre+, Maastricht, all outside the submitted work; MS coauthored the editorial on the Netherlands 35