An imputation study shows that missing outcome data can substantially bias pooled estimates in systematic reviews of patient-reported outcomes

Abstract

Background and Objectives Missing outcome data (hereafter referred to as “missing data,” typically due to loss to follow-up) is a major problem in randomized controlled trials (RCTs) and systematic reviews of RCTs. While prior work has examined the impact of missing binary outcomes, the influence of missing continuous patient-reported outcome measures (PROMs) on pooled effect estimates remains poorly understood. We therefore assessed the risk of bias introduced by missing data in systematic reviews of PROMs. Study Design and Setting We selected a representative sample of 100 systematic reviews that included meta-analyses reporting a statistically significant effect on a continuous patient-reported efficacy outcome. We applied four increasingly stringent imputation strategies based on the grading of recommendations assessment, development, and evaluation (GRADE) approach, along with three alternative approaches for handling studies in which investigators had already imputed results for missing data. We also conducted Firth logistic regression analyses to identify factors associated with crossing the null after imputation. Results Results from 100 systematic reviews that included 1298 RCTs proved similar across all three approaches to addressing imputed data. Using the least stringent strategy for imputing missing data, the percentage of meta-analyses in which the 95% CI crossed the null proved under 4%. Applying the next most stringent strategy, the percentage of CIs that crossed the null increased to 47.9%. Percentages crossing the null increased only marginally for the two most stringent approaches, crossing up to 53.1% in the next most stringent and 54.2% in the most stringent. Firth logistic regression identified two significant predictors of crossing the null after imputation: a higher average missing data (odds ratio [OR] 1.23, 95% CI: 1.11–1.43 per 1% increase in missing data) and a larger magnitude of the treatment effect, which was associated with lower odds of crossing the null (OR 0.70, 95% CI: 0.39–0.91 per 1 standardized mean difference increase). Neither database type (Cochrane vs. non-Cochrane) nor duration of follow-up proved associated with CI crossing the null. Conclusion A plausible imputation approach to test the potential risk of bias as a result of missing data in studies addressing treatment effects on PROMs resulted in 95% CIs in a high proportion of studies initially suggesting benefit crossing the null. The greater the proportion of missing data and the smaller the treatment effect, the more likely the CI crossed the null. Systematic review authors may consider formally testing the robustness of their results with respect to missing data. Plain Language Summary When studies included in a systematic review have missing outcome data, the study results may be biased and therefore misleading. If there is not much missing data, this is not a problem. If there is lots of missing data, it can be a big problem. Researchers have suggested ways of highlighting how much of a problem missing data represents. This study compared four methods for handling missing data in continuous outcomes that measures patients’ experience, we call these patient-reported outcomes. We found that reasonable approaches to assessing the possible bias from missing data in systematic reviews and meta-analyses frequently highlight substantial problems and suggest that more guarded conclusions may be warranted in systematic reviews. These findings highlight the importance of fully considering the amount of missing data when interpreting the results of systematic reviews.