Multimodal personality assessment from audio, visual, and verbal features

The main theme of the present dissertation was the measurement of personality traits through someone’s verbal and non-verbal behavior. In most studies, personality traits were measured using the HEXACO model of personality, a theoretical framework – based on cross-cultural lexical research – that organizes personality using six factors: Honesty-Humility, Emotionality, Extraversion, Agreeableness, Conscientiousness, and Openness to Experience. In one of the studies the Big Five Model was used, which contains similar factors except for Honesty-Humility. Behavior was measured through three modalities: (a) audio, including voice characteristics, such as voice intensity or pitch, (b) visual, including facial expressions and head movements, and (c) verbal, including written or spoken text. All three modalities were automatically extracted using software developed to measure the three types of features at a granular level. Below are presented the main findings across the four empirical chapters of the present dissertation. The findings of the four empirical chapters have significant implications for practitioners and personality psychologists, alike. Regarding practitioners (e.g., AVI vendors), the results suggest that the content of job interview questions should be carefully designed to activate the traits someone is interested in measuring. The more the content of the interview questions aligns with the constructs to-be-measured (e.g., personality traits), the more the behaviors exhibited in those questions will correlate with the constructs of interest. Furthermore, even though the algorithm in Chapter 4 was relatively free of biases, some biases did emerge (e.g., existing gender differences were sometimes further exacerbated). As a result, practitioners might consider applying bias mitigation techniques when employing AVIs in selection contexts, even though such techniques might reduce the overall performance of machine learning models. Regarding personality psychologists, these results suggest that personality inferences are mainly driven by verbal instead of non-verbal behaviors. The kernel of truth in text-based personality assessment further highlighted the linguistic behaviors that contribute the most in accurate personality assessment. Finally, the results showed that the asymmetry in explained variance between self- and observer reports was accounted for by the level of contextualization of personality assessment, as suggested by the bandwidth-fidelity dilemma. This suggests, that contextualization of personality assessment seems to be the main explanation of the asymmetry, and theoretical frameworks on the accuracy of personality assessment, such as the SOKA model, might need to integrate contextualization as an important component to explain the asymmetry.