Toward an Objective Assessment of Implementation Processes for Innovations in Health Care: Psychometric Evaluation of the Normalization Measure Development (NoMAD) Questionnaire Among Mental Health Care Professionals

Christiaan Vis, Jeroen Ruwaard, Tracy Finch, Tim Rapley, Derek de Beurs, Henk van Stel, Britt van Lettow, Mayke Mol, Annet Kleiboer, Heleen Riper, Jan Smit

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

BACKGROUND: Successfully implementing eMental health (eMH) interventions in routine mental health care constitutes a major challenge. Reliable instruments to assess implementation progress are essential. The Normalization MeAsure Development (NoMAD) study developed a brief self-report questionnaire that could be helpful in measuring implementation progress. Based on the Normalization Process Theory, this instrument focuses on 4 generative mechanisms involved in implementation processes: coherence, cognitive participation, collective action, and reflexive monitoring. OBJECTIVE: The aim of this study was to translate the NoMAD questionnaire to Dutch and to confirm the factor structure in Dutch mental health care settings. METHODS: Dutch mental health care professionals involved in eMH implementation were invited to complete the translated NoMAD questionnaire. Confirmatory factor analysis (CFA) was conducted to verify interpretability of scale scores for 3 models: (1) the theoretical 4-factor structure, (2) a unidimensional model, and (3) a hierarchical model. Potential improvements were explored, and correlated scale scores with 3 control questions were used to assess convergent validity. RESULTS: A total of 262 professionals from mental health care settings in the Netherlands completed the questionnaire (female: 81.7%; mean age: 45 [SD=11]). The internal consistency of the 20-item questionnaire was acceptable (.62≤alpha≤.85). The theorized 4-factor model fitted the data slightly better in the CFA than the hierarchical model (Comparative Fit Index=0.90, Tucker Lewis Index=0.88, Root Mean Square Error of Approximation=0.10, Standardized Root Mean Square Residual=0.12, χ22=22.5, P≤.05). However, the difference is small and possibly not outweighing the practical relevance of a total score and subscale scores combined in one hierarchical model. One item was identified as weak (λCA.2=0.10). A moderate-to-strong convergent validity with 3 control questions was found for the Collective Participation scale (.47≤r≤.54, P≤.05). CONCLUSIONS: NoMAD's theoretical factor structure was confirmed in Dutch mental health settings to acceptable standards but with room for improvement. The hierarchical model might prove useful in increasing the practical utility of the NoMAD questionnaire by combining a total score with information on the 4 generative mechanisms. Future research should assess the predictive value and responsiveness over time and elucidate the conceptual interpretability of NoMAD in eMH implementation practices.

Original languageEnglish
Pages (from-to)e12376
JournalJournal of Medical Internet Research
Volume21
Issue number2
DOIs
Publication statusPublished - 20 Feb 2019

Fingerprint

Process Assessment (Health Care)
Psychometrics
Mental Health
Delivery of Health Care
Statistical Factor Analysis
Health
Netherlands
Self Report
Surveys and Questionnaires
Theoretical Models

Keywords

  • eHealth
  • eMental health
  • implementation assessment
  • implementation science
  • normalization process theory
  • psychometrics

Cite this

@article{94e5dedd5cce44f385e587a6c557ecdd,
title = "Toward an Objective Assessment of Implementation Processes for Innovations in Health Care: Psychometric Evaluation of the Normalization Measure Development (NoMAD) Questionnaire Among Mental Health Care Professionals",
abstract = "BACKGROUND: Successfully implementing eMental health (eMH) interventions in routine mental health care constitutes a major challenge. Reliable instruments to assess implementation progress are essential. The Normalization MeAsure Development (NoMAD) study developed a brief self-report questionnaire that could be helpful in measuring implementation progress. Based on the Normalization Process Theory, this instrument focuses on 4 generative mechanisms involved in implementation processes: coherence, cognitive participation, collective action, and reflexive monitoring. OBJECTIVE: The aim of this study was to translate the NoMAD questionnaire to Dutch and to confirm the factor structure in Dutch mental health care settings. METHODS: Dutch mental health care professionals involved in eMH implementation were invited to complete the translated NoMAD questionnaire. Confirmatory factor analysis (CFA) was conducted to verify interpretability of scale scores for 3 models: (1) the theoretical 4-factor structure, (2) a unidimensional model, and (3) a hierarchical model. Potential improvements were explored, and correlated scale scores with 3 control questions were used to assess convergent validity. RESULTS: A total of 262 professionals from mental health care settings in the Netherlands completed the questionnaire (female: 81.7{\%}; mean age: 45 [SD=11]). The internal consistency of the 20-item questionnaire was acceptable (.62≤alpha≤.85). The theorized 4-factor model fitted the data slightly better in the CFA than the hierarchical model (Comparative Fit Index=0.90, Tucker Lewis Index=0.88, Root Mean Square Error of Approximation=0.10, Standardized Root Mean Square Residual=0.12, χ22=22.5, P≤.05). However, the difference is small and possibly not outweighing the practical relevance of a total score and subscale scores combined in one hierarchical model. One item was identified as weak (λCA.2=0.10). A moderate-to-strong convergent validity with 3 control questions was found for the Collective Participation scale (.47≤r≤.54, P≤.05). CONCLUSIONS: NoMAD's theoretical factor structure was confirmed in Dutch mental health settings to acceptable standards but with room for improvement. The hierarchical model might prove useful in increasing the practical utility of the NoMAD questionnaire by combining a total score with information on the 4 generative mechanisms. Future research should assess the predictive value and responsiveness over time and elucidate the conceptual interpretability of NoMAD in eMH implementation practices.",
keywords = "eHealth, eMental health, implementation assessment, implementation science, normalization process theory, psychometrics",
author = "Christiaan Vis and Jeroen Ruwaard and Tracy Finch and Tim Rapley and {de Beurs}, Derek and {van Stel}, Henk and {van Lettow}, Britt and Mayke Mol and Annet Kleiboer and Heleen Riper and Jan Smit",
year = "2019",
month = "2",
day = "20",
doi = "10.2196/12376",
language = "English",
volume = "21",
pages = "e12376",
journal = "Journal of Medical Internet Research",
issn = "1438-8871",
publisher = "Journal of medical Internet Research",
number = "2",

}

Toward an Objective Assessment of Implementation Processes for Innovations in Health Care : Psychometric Evaluation of the Normalization Measure Development (NoMAD) Questionnaire Among Mental Health Care Professionals. / Vis, Christiaan; Ruwaard, Jeroen; Finch, Tracy; Rapley, Tim; de Beurs, Derek; van Stel, Henk; van Lettow, Britt; Mol, Mayke; Kleiboer, Annet; Riper, Heleen; Smit, Jan.

In: Journal of Medical Internet Research, Vol. 21, No. 2, 20.02.2019, p. e12376.

Research output: Contribution to JournalArticleAcademicpeer-review

TY - JOUR

T1 - Toward an Objective Assessment of Implementation Processes for Innovations in Health Care

T2 - Psychometric Evaluation of the Normalization Measure Development (NoMAD) Questionnaire Among Mental Health Care Professionals

AU - Vis, Christiaan

AU - Ruwaard, Jeroen

AU - Finch, Tracy

AU - Rapley, Tim

AU - de Beurs, Derek

AU - van Stel, Henk

AU - van Lettow, Britt

AU - Mol, Mayke

AU - Kleiboer, Annet

AU - Riper, Heleen

AU - Smit, Jan

PY - 2019/2/20

Y1 - 2019/2/20

N2 - BACKGROUND: Successfully implementing eMental health (eMH) interventions in routine mental health care constitutes a major challenge. Reliable instruments to assess implementation progress are essential. The Normalization MeAsure Development (NoMAD) study developed a brief self-report questionnaire that could be helpful in measuring implementation progress. Based on the Normalization Process Theory, this instrument focuses on 4 generative mechanisms involved in implementation processes: coherence, cognitive participation, collective action, and reflexive monitoring. OBJECTIVE: The aim of this study was to translate the NoMAD questionnaire to Dutch and to confirm the factor structure in Dutch mental health care settings. METHODS: Dutch mental health care professionals involved in eMH implementation were invited to complete the translated NoMAD questionnaire. Confirmatory factor analysis (CFA) was conducted to verify interpretability of scale scores for 3 models: (1) the theoretical 4-factor structure, (2) a unidimensional model, and (3) a hierarchical model. Potential improvements were explored, and correlated scale scores with 3 control questions were used to assess convergent validity. RESULTS: A total of 262 professionals from mental health care settings in the Netherlands completed the questionnaire (female: 81.7%; mean age: 45 [SD=11]). The internal consistency of the 20-item questionnaire was acceptable (.62≤alpha≤.85). The theorized 4-factor model fitted the data slightly better in the CFA than the hierarchical model (Comparative Fit Index=0.90, Tucker Lewis Index=0.88, Root Mean Square Error of Approximation=0.10, Standardized Root Mean Square Residual=0.12, χ22=22.5, P≤.05). However, the difference is small and possibly not outweighing the practical relevance of a total score and subscale scores combined in one hierarchical model. One item was identified as weak (λCA.2=0.10). A moderate-to-strong convergent validity with 3 control questions was found for the Collective Participation scale (.47≤r≤.54, P≤.05). CONCLUSIONS: NoMAD's theoretical factor structure was confirmed in Dutch mental health settings to acceptable standards but with room for improvement. The hierarchical model might prove useful in increasing the practical utility of the NoMAD questionnaire by combining a total score with information on the 4 generative mechanisms. Future research should assess the predictive value and responsiveness over time and elucidate the conceptual interpretability of NoMAD in eMH implementation practices.

AB - BACKGROUND: Successfully implementing eMental health (eMH) interventions in routine mental health care constitutes a major challenge. Reliable instruments to assess implementation progress are essential. The Normalization MeAsure Development (NoMAD) study developed a brief self-report questionnaire that could be helpful in measuring implementation progress. Based on the Normalization Process Theory, this instrument focuses on 4 generative mechanisms involved in implementation processes: coherence, cognitive participation, collective action, and reflexive monitoring. OBJECTIVE: The aim of this study was to translate the NoMAD questionnaire to Dutch and to confirm the factor structure in Dutch mental health care settings. METHODS: Dutch mental health care professionals involved in eMH implementation were invited to complete the translated NoMAD questionnaire. Confirmatory factor analysis (CFA) was conducted to verify interpretability of scale scores for 3 models: (1) the theoretical 4-factor structure, (2) a unidimensional model, and (3) a hierarchical model. Potential improvements were explored, and correlated scale scores with 3 control questions were used to assess convergent validity. RESULTS: A total of 262 professionals from mental health care settings in the Netherlands completed the questionnaire (female: 81.7%; mean age: 45 [SD=11]). The internal consistency of the 20-item questionnaire was acceptable (.62≤alpha≤.85). The theorized 4-factor model fitted the data slightly better in the CFA than the hierarchical model (Comparative Fit Index=0.90, Tucker Lewis Index=0.88, Root Mean Square Error of Approximation=0.10, Standardized Root Mean Square Residual=0.12, χ22=22.5, P≤.05). However, the difference is small and possibly not outweighing the practical relevance of a total score and subscale scores combined in one hierarchical model. One item was identified as weak (λCA.2=0.10). A moderate-to-strong convergent validity with 3 control questions was found for the Collective Participation scale (.47≤r≤.54, P≤.05). CONCLUSIONS: NoMAD's theoretical factor structure was confirmed in Dutch mental health settings to acceptable standards but with room for improvement. The hierarchical model might prove useful in increasing the practical utility of the NoMAD questionnaire by combining a total score with information on the 4 generative mechanisms. Future research should assess the predictive value and responsiveness over time and elucidate the conceptual interpretability of NoMAD in eMH implementation practices.

KW - eHealth

KW - eMental health

KW - implementation assessment

KW - implementation science

KW - normalization process theory

KW - psychometrics

UR - http://www.scopus.com/inward/record.url?scp=85061851790&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061851790&partnerID=8YFLogxK

U2 - 10.2196/12376

DO - 10.2196/12376

M3 - Article

VL - 21

SP - e12376

JO - Journal of Medical Internet Research

JF - Journal of Medical Internet Research

SN - 1438-8871

IS - 2

ER -