Constructed response formats and their effects on minority-majority differences and validity

Filip Lievens, Paul R. Sackett, Jeffrey A. Dahlke, Janneke K. Oostrom, Britt De Soete

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

The inflow of immigrants challenges organizations to consider alternative selection procedures that reduce potential minority (immigrants)-majority (natives) differences, while maintaining valid predictions of performance. To deal with this challenge, this paper proposes response format as a practically and theoretically relevant factor for situational judgment tests (SJTs). We examine a range of response format categories (from traditional multiple-choice formats to more innovative constructed response formats) and conceptually link these response formats to mechanisms underlying minority-majority differences. Two field experiments are conducted with SJTs. Study 1 (274 job seekers) contrasts minority-majority differences in scores on a multiple-choice versus a written constructed response format. Written constructed responses produce much smaller minority-majority differences (d = .28 vs. d = .92). In Study 2 (269 incumbents), scores on a written constructed versus an audiovisual constructed format are compared. The audiovisual format further reduces minority-majority differences (d = .09 vs. d = .41), with validities remaining the same. Results are suggestive of cognitive load as a contributor to the reduction in minority-majority differences, as are rater effects: Scores of raters evaluating transcribed audiovisual responses, which anonymized test takers, produce larger differences. In sum, altering response modality via more realistic response formats (i.e., the audiovisual constructed format) leads to significant reductions in minority-majority differences without impairing criterion-related validity. Implications for selection theory and practice are discussed.

Original languageEnglish
Pages (from-to)715-726
Number of pages12
JournalJournal of Applied Psychology
Volume104
Issue number5
DOIs
Publication statusPublished - 1 May 2019

Fingerprint

Population Groups
Organizations

Keywords

  • Immigrants
  • Minorities
  • Response format
  • Situational judgment tests
  • Subgroup differences

Cite this

Lievens, Filip ; Sackett, Paul R. ; Dahlke, Jeffrey A. ; Oostrom, Janneke K. ; Soete, Britt De. / Constructed response formats and their effects on minority-majority differences and validity. In: Journal of Applied Psychology. 2019 ; Vol. 104, No. 5. pp. 715-726.
@article{42d0c24094a44ce6a9b44ac92cd61e9b,
title = "Constructed response formats and their effects on minority-majority differences and validity",
abstract = "The inflow of immigrants challenges organizations to consider alternative selection procedures that reduce potential minority (immigrants)-majority (natives) differences, while maintaining valid predictions of performance. To deal with this challenge, this paper proposes response format as a practically and theoretically relevant factor for situational judgment tests (SJTs). We examine a range of response format categories (from traditional multiple-choice formats to more innovative constructed response formats) and conceptually link these response formats to mechanisms underlying minority-majority differences. Two field experiments are conducted with SJTs. Study 1 (274 job seekers) contrasts minority-majority differences in scores on a multiple-choice versus a written constructed response format. Written constructed responses produce much smaller minority-majority differences (d = .28 vs. d = .92). In Study 2 (269 incumbents), scores on a written constructed versus an audiovisual constructed format are compared. The audiovisual format further reduces minority-majority differences (d = .09 vs. d = .41), with validities remaining the same. Results are suggestive of cognitive load as a contributor to the reduction in minority-majority differences, as are rater effects: Scores of raters evaluating transcribed audiovisual responses, which anonymized test takers, produce larger differences. In sum, altering response modality via more realistic response formats (i.e., the audiovisual constructed format) leads to significant reductions in minority-majority differences without impairing criterion-related validity. Implications for selection theory and practice are discussed.",
keywords = "Immigrants, Minorities, Response format, Situational judgment tests, Subgroup differences",
author = "Filip Lievens and Sackett, {Paul R.} and Dahlke, {Jeffrey A.} and Oostrom, {Janneke K.} and Soete, {Britt De}",
year = "2019",
month = "5",
day = "1",
doi = "10.1037/apl0000367",
language = "English",
volume = "104",
pages = "715--726",
journal = "Journal of Applied Psychology",
issn = "0021-9010",
publisher = "American Psychological Association",
number = "5",

}

Constructed response formats and their effects on minority-majority differences and validity. / Lievens, Filip; Sackett, Paul R.; Dahlke, Jeffrey A.; Oostrom, Janneke K.; Soete, Britt De.

In: Journal of Applied Psychology, Vol. 104, No. 5, 01.05.2019, p. 715-726.

Research output: Contribution to JournalArticleAcademicpeer-review

TY - JOUR

T1 - Constructed response formats and their effects on minority-majority differences and validity

AU - Lievens, Filip

AU - Sackett, Paul R.

AU - Dahlke, Jeffrey A.

AU - Oostrom, Janneke K.

AU - Soete, Britt De

PY - 2019/5/1

Y1 - 2019/5/1

N2 - The inflow of immigrants challenges organizations to consider alternative selection procedures that reduce potential minority (immigrants)-majority (natives) differences, while maintaining valid predictions of performance. To deal with this challenge, this paper proposes response format as a practically and theoretically relevant factor for situational judgment tests (SJTs). We examine a range of response format categories (from traditional multiple-choice formats to more innovative constructed response formats) and conceptually link these response formats to mechanisms underlying minority-majority differences. Two field experiments are conducted with SJTs. Study 1 (274 job seekers) contrasts minority-majority differences in scores on a multiple-choice versus a written constructed response format. Written constructed responses produce much smaller minority-majority differences (d = .28 vs. d = .92). In Study 2 (269 incumbents), scores on a written constructed versus an audiovisual constructed format are compared. The audiovisual format further reduces minority-majority differences (d = .09 vs. d = .41), with validities remaining the same. Results are suggestive of cognitive load as a contributor to the reduction in minority-majority differences, as are rater effects: Scores of raters evaluating transcribed audiovisual responses, which anonymized test takers, produce larger differences. In sum, altering response modality via more realistic response formats (i.e., the audiovisual constructed format) leads to significant reductions in minority-majority differences without impairing criterion-related validity. Implications for selection theory and practice are discussed.

AB - The inflow of immigrants challenges organizations to consider alternative selection procedures that reduce potential minority (immigrants)-majority (natives) differences, while maintaining valid predictions of performance. To deal with this challenge, this paper proposes response format as a practically and theoretically relevant factor for situational judgment tests (SJTs). We examine a range of response format categories (from traditional multiple-choice formats to more innovative constructed response formats) and conceptually link these response formats to mechanisms underlying minority-majority differences. Two field experiments are conducted with SJTs. Study 1 (274 job seekers) contrasts minority-majority differences in scores on a multiple-choice versus a written constructed response format. Written constructed responses produce much smaller minority-majority differences (d = .28 vs. d = .92). In Study 2 (269 incumbents), scores on a written constructed versus an audiovisual constructed format are compared. The audiovisual format further reduces minority-majority differences (d = .09 vs. d = .41), with validities remaining the same. Results are suggestive of cognitive load as a contributor to the reduction in minority-majority differences, as are rater effects: Scores of raters evaluating transcribed audiovisual responses, which anonymized test takers, produce larger differences. In sum, altering response modality via more realistic response formats (i.e., the audiovisual constructed format) leads to significant reductions in minority-majority differences without impairing criterion-related validity. Implications for selection theory and practice are discussed.

KW - Immigrants

KW - Minorities

KW - Response format

KW - Situational judgment tests

KW - Subgroup differences

UR - http://www.scopus.com/inward/record.url?scp=85056522452&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056522452&partnerID=8YFLogxK

U2 - 10.1037/apl0000367

DO - 10.1037/apl0000367

M3 - Article

VL - 104

SP - 715

EP - 726

JO - Journal of Applied Psychology

JF - Journal of Applied Psychology

SN - 0021-9010

IS - 5

ER -