Bias through selective inclusion and attrition: Representativeness when comparing provider performance with routine outcome monitoring data

Edwin de Beurs, Lisanne Warmerdam, Jos Twisk

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

Background: Observational research based on routine outcome monitoring is prone to missing data, and outcomes can be biased due to selective inclusion at baseline or selective attrition at posttest. As patients with complete data may not be representative of all patients of a provider, missing data may bias results, especially when missingness is not random but systematic. Methods: The present study establishes clinical and demographic patient variables relevant for representativeness of the outcome information. It applies strategies to estimate sample selection bias (weighting by inclusion propensity) and selective attrition bias (multiple imputation based on multilevel regression analysis) and estimates the extent of their impact on an index of provider performance. The association between estimated bias and response rate is also investigated. Results: Provider-based analyses showed that in current practice, the effect of selective inclusion was minimal, but attrition had a more substantial effect, biasing results in both directions: overstating and understating performance. For 22% of the providers, attrition bias was estimated to be in excess of 0.05 ES. Bias was associated with overall response rate (r =.50). When selective inclusion and attrition bring providers' response below 50%, it is more likely that selection bias increased beyond a critical level, and conclusions on the comparative performance of such providers may be misleading. Conclusions: Estimates of provider performance were biased by selection, especially by missing data at posttest. Results on the extent and direction of bias and minimal requirements for response rates to arrive at unbiased performance indicators are discussed.

Original languageEnglish
Pages (from-to)430-439
Number of pages10
JournalClinical Psychology and Psychotherapy
Volume26
Issue number4
Early online date18 Mar 2019
DOIs
Publication statusPublished - Jul 2019
Externally publishedYes

Fingerprint

Selection Bias
Patient Advocacy
Multilevel Analysis
Regression Analysis
Demography
Research
Direction compound
Clinical Studies

Keywords

  • attrition bias
  • imputation
  • propensity weighting
  • response rate
  • selection bias

Cite this

@article{27cc1c1d67044ef193dd6a7082503809,
title = "Bias through selective inclusion and attrition: Representativeness when comparing provider performance with routine outcome monitoring data",
abstract = "Background: Observational research based on routine outcome monitoring is prone to missing data, and outcomes can be biased due to selective inclusion at baseline or selective attrition at posttest. As patients with complete data may not be representative of all patients of a provider, missing data may bias results, especially when missingness is not random but systematic. Methods: The present study establishes clinical and demographic patient variables relevant for representativeness of the outcome information. It applies strategies to estimate sample selection bias (weighting by inclusion propensity) and selective attrition bias (multiple imputation based on multilevel regression analysis) and estimates the extent of their impact on an index of provider performance. The association between estimated bias and response rate is also investigated. Results: Provider-based analyses showed that in current practice, the effect of selective inclusion was minimal, but attrition had a more substantial effect, biasing results in both directions: overstating and understating performance. For 22{\%} of the providers, attrition bias was estimated to be in excess of 0.05 ES. Bias was associated with overall response rate (r =.50). When selective inclusion and attrition bring providers' response below 50{\%}, it is more likely that selection bias increased beyond a critical level, and conclusions on the comparative performance of such providers may be misleading. Conclusions: Estimates of provider performance were biased by selection, especially by missing data at posttest. Results on the extent and direction of bias and minimal requirements for response rates to arrive at unbiased performance indicators are discussed.",
keywords = "attrition bias, imputation, propensity weighting, response rate, selection bias",
author = "{de Beurs}, Edwin and Lisanne Warmerdam and Jos Twisk",
year = "2019",
month = "7",
doi = "10.1002/cpp.2364",
language = "English",
volume = "26",
pages = "430--439",
journal = "Clinical Psychology and Psychotherapy",
issn = "1063-3995",
publisher = "John Wiley and Sons Ltd",
number = "4",

}

Bias through selective inclusion and attrition : Representativeness when comparing provider performance with routine outcome monitoring data. / de Beurs, Edwin; Warmerdam, Lisanne; Twisk, Jos.

In: Clinical Psychology and Psychotherapy, Vol. 26, No. 4, 07.2019, p. 430-439.

Research output: Contribution to JournalArticleAcademicpeer-review

TY - JOUR

T1 - Bias through selective inclusion and attrition

T2 - Representativeness when comparing provider performance with routine outcome monitoring data

AU - de Beurs, Edwin

AU - Warmerdam, Lisanne

AU - Twisk, Jos

PY - 2019/7

Y1 - 2019/7

N2 - Background: Observational research based on routine outcome monitoring is prone to missing data, and outcomes can be biased due to selective inclusion at baseline or selective attrition at posttest. As patients with complete data may not be representative of all patients of a provider, missing data may bias results, especially when missingness is not random but systematic. Methods: The present study establishes clinical and demographic patient variables relevant for representativeness of the outcome information. It applies strategies to estimate sample selection bias (weighting by inclusion propensity) and selective attrition bias (multiple imputation based on multilevel regression analysis) and estimates the extent of their impact on an index of provider performance. The association between estimated bias and response rate is also investigated. Results: Provider-based analyses showed that in current practice, the effect of selective inclusion was minimal, but attrition had a more substantial effect, biasing results in both directions: overstating and understating performance. For 22% of the providers, attrition bias was estimated to be in excess of 0.05 ES. Bias was associated with overall response rate (r =.50). When selective inclusion and attrition bring providers' response below 50%, it is more likely that selection bias increased beyond a critical level, and conclusions on the comparative performance of such providers may be misleading. Conclusions: Estimates of provider performance were biased by selection, especially by missing data at posttest. Results on the extent and direction of bias and minimal requirements for response rates to arrive at unbiased performance indicators are discussed.

AB - Background: Observational research based on routine outcome monitoring is prone to missing data, and outcomes can be biased due to selective inclusion at baseline or selective attrition at posttest. As patients with complete data may not be representative of all patients of a provider, missing data may bias results, especially when missingness is not random but systematic. Methods: The present study establishes clinical and demographic patient variables relevant for representativeness of the outcome information. It applies strategies to estimate sample selection bias (weighting by inclusion propensity) and selective attrition bias (multiple imputation based on multilevel regression analysis) and estimates the extent of their impact on an index of provider performance. The association between estimated bias and response rate is also investigated. Results: Provider-based analyses showed that in current practice, the effect of selective inclusion was minimal, but attrition had a more substantial effect, biasing results in both directions: overstating and understating performance. For 22% of the providers, attrition bias was estimated to be in excess of 0.05 ES. Bias was associated with overall response rate (r =.50). When selective inclusion and attrition bring providers' response below 50%, it is more likely that selection bias increased beyond a critical level, and conclusions on the comparative performance of such providers may be misleading. Conclusions: Estimates of provider performance were biased by selection, especially by missing data at posttest. Results on the extent and direction of bias and minimal requirements for response rates to arrive at unbiased performance indicators are discussed.

KW - attrition bias

KW - imputation

KW - propensity weighting

KW - response rate

KW - selection bias

UR - http://www.scopus.com/inward/record.url?scp=85065036504&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065036504&partnerID=8YFLogxK

U2 - 10.1002/cpp.2364

DO - 10.1002/cpp.2364

M3 - Article

VL - 26

SP - 430

EP - 439

JO - Clinical Psychology and Psychotherapy

JF - Clinical Psychology and Psychotherapy

SN - 1063-3995

IS - 4

ER -