How linkage error affects hidden Markov model estimates: A sensitivity analysis

P.K.P. Pankowska, Bart F.M. Bakker, D.L. Oberski, D. Pavlopoulos

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

Hidden Markov models (HMMs) are increasingly used to estimate and correct for classification error in categorical, longitudinal data, without the need for a “gold standard,” error-free data source. To accomplish this, HMMs require multiple observations over time on a single indicator and assume that the errors in these indicators are conditionally independent. Unfortunately, this “local independence” assumption is often unrealistic, untestable, and a source of serious bias. Linking independent data sources can solve this problem by making the local independence assumption plausible across sources, while potentially allowing for local dependence within sources. However, record linkage introduces a new problem: the records may be erroneously linked or incorrectly not linked. In this paper, we investigate the effects of linkage error on HMM estimates of transitions between employment contract types. Our data come from linking a labor force survey to administrative employer records; this linkage yields two indicators per time point that are plausibly conditionally independent. Our results indicate that both false-negative and false-positive linkage error turn out to be problematic primarily if the error is large and highly correlated with the dependent variable. Moreover, under certain conditions, false-positive linkage error (mislinkage) in fact acts as another source of misclassification that the HMM can absorb into its error-rate estimates, leaving the latent transition estimates unbiased. In these cases, measurement error modeling already accounts for linkage error. Our results also indicate where these conditions break down and more complex methods would be needed.

Original languageEnglish
Pages (from-to)483-512
Number of pages30
JournalJournal of Survey Statistics and Methodology
Volume8
Issue number3
Early online date29 May 2019
DOIs
Publication statusPublished - Jun 2020

Keywords

  • Classification error
  • Hidden Markov model (HMM)
  • Linkage error
  • Measurement error
  • Misclassification
  • Record linkage

Fingerprint

Dive into the research topics of 'How linkage error affects hidden Markov model estimates: A sensitivity analysis'. Together they form a unique fingerprint.

Cite this