Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment

Merle M. Reimann; Florian A. Kunneman; Catharine Oertel; Koen V. Hindriks

doi:10.1145/3719160.3736629

Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment

Merle M. Reimann^*
, Florian A. Kunneman
, Catharine Oertel
, Koen V. Hindriks

^*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

When a user interacts with a conversational agent for the first time, they may not be aware of the agent’s capabilities, leading to suboptimal use or interaction breakdowns. To avoid a mismatch with the actual capabilities, the agent’s capabilities have to be made transparent to the user. To investigate whether communication of an agent’s capabilities during interactions enhances transparency and improves the user’s mental model, we conducted a user study with 56 participants. Each participant had three speech-based interactions with an agent that communicated its capabilities or an agent that did not. Our results suggest that the communication led to a change in user behavior with significantly longer utterances. However, the users’ mental models of the agent’s capabilities were not significantly different between the conditions. Participants were able to significantly improve their knowledge of the agent’s capabilities by aligning their mental model over time in both conditions.

Original language	English
Title of host publication	CUI '25
Subtitle of host publication	Proceedings of the 7th ACM Conference on Conversational User Interfaces
Editors	Jaisie Sin, Edith Law, Jim Wallace, Cosmin Munteanu, Danai Korre
Publisher	Association for Computing Machinery, Inc
Pages	1-12
Number of pages	12
ISBN (Electronic)	9798400715273
DOIs	https://doi.org/10.1145/3719160.3736629
Publication status	Published - 2025
Event	7th Conference on Conversational User Interfaces, CUI 2025 - Waterloo, Canada Duration: 8 Jul 2025 → 10 Jul 2025

Conference

Conference	7th Conference on Conversational User Interfaces, CUI 2025
Country/Territory	Canada
City	Waterloo
Period	8/07/25 → 10/07/25

Bibliographical note

Publisher Copyright:
© 2025 Copyright held by the owner/author(s).

Keywords

Capability Communication
Conversational Agents
Dialogue Management
Mental Models
Transparency

Access to Document

10.1145/3719160.3736629Licence: CC BY

Persistent URL (handle)

Persistent link

Cite this

Reimann, M. M., Kunneman, F. A., Oertel, C., & Hindriks, K. V. (2025). Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment. In J. Sin, E. Law, J. Wallace, C. Munteanu, & D. Korre (Eds.), CUI '25: Proceedings of the 7th ACM Conference on Conversational User Interfaces (pp. 1-12). Article 48 Association for Computing Machinery, Inc. https://doi.org/10.1145/3719160.3736629

Reimann, Merle M. ; Kunneman, Florian A. ; Oertel, Catharine et al. / Transparent Conversational Agents : The Impact of Capability Communication on User Behavior and Mental Model Alignment. CUI '25: Proceedings of the 7th ACM Conference on Conversational User Interfaces. editor / Jaisie Sin ; Edith Law ; Jim Wallace ; Cosmin Munteanu ; Danai Korre. Association for Computing Machinery, Inc, 2025. pp. 1-12

@inproceedings{2e61b6338c88418e8d62c97c7af40b05,

title = "Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment",

abstract = "When a user interacts with a conversational agent for the first time, they may not be aware of the agent{\textquoteright}s capabilities, leading to suboptimal use or interaction breakdowns. To avoid a mismatch with the actual capabilities, the agent{\textquoteright}s capabilities have to be made transparent to the user. To investigate whether communication of an agent{\textquoteright}s capabilities during interactions enhances transparency and improves the user{\textquoteright}s mental model, we conducted a user study with 56 participants. Each participant had three speech-based interactions with an agent that communicated its capabilities or an agent that did not. Our results suggest that the communication led to a change in user behavior with significantly longer utterances. However, the users{\textquoteright} mental models of the agent{\textquoteright}s capabilities were not significantly different between the conditions. Participants were able to significantly improve their knowledge of the agent{\textquoteright}s capabilities by aligning their mental model over time in both conditions.",

keywords = "Capability Communication, Conversational Agents, Dialogue Management, Mental Models, Transparency",

author = "Reimann, \{Merle M.\} and Kunneman, \{Florian A.\} and Catharine Oertel and Hindriks, \{Koen V.\}",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 7th Conference on Conversational User Interfaces, CUI 2025 ; Conference date: 08-07-2025 Through 10-07-2025",

year = "2025",

doi = "10.1145/3719160.3736629",

language = "English",

pages = "1--12",

editor = "Jaisie Sin and Edith Law and Jim Wallace and Cosmin Munteanu and Danai Korre",

booktitle = "CUI '25",

publisher = "Association for Computing Machinery, Inc",

}

Reimann, MM, Kunneman, FA, Oertel, C & Hindriks, KV 2025, Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment. in J Sin, E Law, J Wallace, C Munteanu & D Korre (eds), CUI '25: Proceedings of the 7th ACM Conference on Conversational User Interfaces., 48, Association for Computing Machinery, Inc, pp. 1-12, 7th Conference on Conversational User Interfaces, CUI 2025, Waterloo, Canada, 8/07/25. https://doi.org/10.1145/3719160.3736629

Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment. / Reimann, Merle M.; Kunneman, Florian A.; Oertel, Catharine et al.
CUI '25: Proceedings of the 7th ACM Conference on Conversational User Interfaces. ed. / Jaisie Sin; Edith Law; Jim Wallace; Cosmin Munteanu; Danai Korre. Association for Computing Machinery, Inc, 2025. p. 1-12 48.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Transparent Conversational Agents

T2 - 7th Conference on Conversational User Interfaces, CUI 2025

AU - Reimann, Merle M.

AU - Kunneman, Florian A.

AU - Oertel, Catharine

AU - Hindriks, Koen V.

PY - 2025

Y1 - 2025

N2 - When a user interacts with a conversational agent for the first time, they may not be aware of the agent’s capabilities, leading to suboptimal use or interaction breakdowns. To avoid a mismatch with the actual capabilities, the agent’s capabilities have to be made transparent to the user. To investigate whether communication of an agent’s capabilities during interactions enhances transparency and improves the user’s mental model, we conducted a user study with 56 participants. Each participant had three speech-based interactions with an agent that communicated its capabilities or an agent that did not. Our results suggest that the communication led to a change in user behavior with significantly longer utterances. However, the users’ mental models of the agent’s capabilities were not significantly different between the conditions. Participants were able to significantly improve their knowledge of the agent’s capabilities by aligning their mental model over time in both conditions.

AB - When a user interacts with a conversational agent for the first time, they may not be aware of the agent’s capabilities, leading to suboptimal use or interaction breakdowns. To avoid a mismatch with the actual capabilities, the agent’s capabilities have to be made transparent to the user. To investigate whether communication of an agent’s capabilities during interactions enhances transparency and improves the user’s mental model, we conducted a user study with 56 participants. Each participant had three speech-based interactions with an agent that communicated its capabilities or an agent that did not. Our results suggest that the communication led to a change in user behavior with significantly longer utterances. However, the users’ mental models of the agent’s capabilities were not significantly different between the conditions. Participants were able to significantly improve their knowledge of the agent’s capabilities by aligning their mental model over time in both conditions.

KW - Capability Communication

KW - Conversational Agents

KW - Dialogue Management

KW - Mental Models

KW - Transparency

UR - https://www.scopus.com/pages/publications/105011589173

UR - https://www.scopus.com/pages/publications/105011589173#tab=citedBy

U2 - 10.1145/3719160.3736629

DO - 10.1145/3719160.3736629

M3 - Conference contribution

AN - SCOPUS:105011589173

SP - 1

EP - 12

BT - CUI '25

A2 - Sin, Jaisie

A2 - Law, Edith

A2 - Wallace, Jim

A2 - Munteanu, Cosmin

A2 - Korre, Danai

PB - Association for Computing Machinery, Inc

Y2 - 8 July 2025 through 10 July 2025

ER -

Reimann MM, Kunneman FA, Oertel C, Hindriks KV. Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment. In Sin J, Law E, Wallace J, Munteanu C, Korre D, editors, CUI '25: Proceedings of the 7th ACM Conference on Conversational User Interfaces. Association for Computing Machinery, Inc. 2025. p. 1-12. 48 Epub 2025 Jul 7. doi: 10.1145/3719160.3736629

Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this