Reconciliation of inconsistent data sources using hidden Markov models

Paulina Karolina Pankowska, Dimitris Pavlopoulos, Bart F.M. Bakker, Daniel Oberski

Research output: Contribution to JournalArticleAcademicpeer-review

128 Downloads (Pure)


This paper discusses how National Statistical Institutes (NSI's) can use hidden Markov models (HMMs) to produce consistent official statistics for categorical, longitudinal variables using inconsistent sources. Two main challenges are addressed: first, the reconciliation of inconsistent sources with multi-indicator HMMs requires linking the sources on the micro level. Such linkage might lead to bias due to linkage error. Second, applying and estimating HMMs regularly is a complicated and expensive procedure. Therefore, it is preferable to use the error parameter estimates as a correction factor for a number of years. However, this might lead to biased structural estimates if measurement error changes over time or if the data collection process changes. Our results on these issues are highly encouraging and imply that the suggested method is appropriate for NSI's. Specifically, linkage error only leads to (substantial) bias in very extreme scenarios. Moreover, measurement error parameters are largely stable over time if no major changes in the data collection process occur. However, when a substantial change in the data collection process occurs, such as a switch from dependent (DI) to independent (INDI) interviewing, re-using measurement error estimates is not advisable.

Original languageEnglish
Pages (from-to)1261-1279
Number of pages19
JournalStatistical Journal of the IAOS
Issue number4
Early online date15 Aug 2020
Publication statusPublished - 25 Nov 2020


  • Data reconciliation
  • dependent interviewing
  • hidden Markov model
  • inconsistent data sources
  • latent class model
  • linkage error
  • measurement error


Dive into the research topics of 'Reconciliation of inconsistent data sources using hidden Markov models'. Together they form a unique fingerprint.

Cite this