Towards Situated AMR: Creating a Corpus of Gesture AMR

Lucia Donatelli; Kenneth Lai; Richard Brutti; James Pustejovsky

doi:10.1007/978-3-031-06018-2_21

Towards Situated AMR: Creating a Corpus of Gesture AMR

Lucia Donatelli
, Kenneth Lai
, Richard Brutti
, James Pustejovsky

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

In this paper, we extend Abstract Meaning Representation (AMR) in order to represent situated multimodal dialogue, with a focus on the modality of gesture. AMR is a general-purpose meaning representation that has become popular for its transparent structure, its ease of annotation and available corpora, and its overall expressiveness. While AMR was designed to represent meaning in language as text or speech, gesture accompanying speech conveys a number of novel communicative dimensions, including situational reference, spatial locations, manner, attitude, orientation, backchanneling, and others. In this paper, we explore how to combine multimodal elements into a single representation for alignment and grounded meaning, using gesture as a case study. As a platform for multimodal situated dialogue annotation, we believe that Gesture AMR has several attractive properties. It is adequately expressive at both utterance and dialogue levels, while easily accommodating the structures inherent in gestural expressions. Further, the native reentrancy facilitates both the linking between modalities and the eventual situational grounding to contextual bindings.

Original language	English
Title of host publication	Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design
Subtitle of host publication	13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part II
Editors	Vincent G. Duffy
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	293-312
Number of pages	20
ISBN (Electronic)	9783031060182
ISBN (Print)	9783031060175
DOIs	https://doi.org/10.1007/978-3-031-06018-2_21
Publication status	Published - 2022
Externally published	Yes
Event	13th International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management, DHM 2022 Held as Part of the 24th HCI International Conference, HCII 2022 - Virtual, Online Duration: 26 Jun 2022 → 1 Jul 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13320 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	13th International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management, DHM 2022 Held as Part of the 24th HCI International Conference, HCII 2022
City	Virtual, Online
Period	26/06/22 → 1/07/22

Funding

This work was supported in part by NSF grant DRL 2019805, to Dr. Pustejovsky at Brandeis University, and an NSF Student Grant to Kenneth Lai, Richard Brutti, and Lucia Donatelli, also funded by NSF grant DRL 2019805. We would like to express our thanks to Nikhil Krishnaswamy for his comments on the multimodal framework motivating the development of Gesture AMR. The views expressed herein are ours alone.

Funders	Funder number
National Science Foundation	DRL 2019805
Brandeis University

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 16 Peace, Justice and Strong Institutions

Access to Document

10.1007/978-3-031-06018-2_21

Persistent URL (handle)

Persistent link

Cite this

Donatelli, L., Lai, K., Brutti, R., & Pustejovsky, J. (2022). Towards Situated AMR: Creating a Corpus of Gesture AMR. In V. G. Duffy (Ed.), Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part II (pp. 293-312). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13320 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-06018-2_21

Donatelli, Lucia ; Lai, Kenneth ; Brutti, Richard et al. / Towards Situated AMR : Creating a Corpus of Gesture AMR. Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part II. editor / Vincent G. Duffy. Springer Science and Business Media Deutschland GmbH, 2022. pp. 293-312 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{6289e834d4564ff3bdf5dc953f39d4ce,

title = "Towards Situated AMR: Creating a Corpus of Gesture AMR",

abstract = "In this paper, we extend Abstract Meaning Representation (AMR) in order to represent situated multimodal dialogue, with a focus on the modality of gesture. AMR is a general-purpose meaning representation that has become popular for its transparent structure, its ease of annotation and available corpora, and its overall expressiveness. While AMR was designed to represent meaning in language as text or speech, gesture accompanying speech conveys a number of novel communicative dimensions, including situational reference, spatial locations, manner, attitude, orientation, backchanneling, and others. In this paper, we explore how to combine multimodal elements into a single representation for alignment and grounded meaning, using gesture as a case study. As a platform for multimodal situated dialogue annotation, we believe that Gesture AMR has several attractive properties. It is adequately expressive at both utterance and dialogue levels, while easily accommodating the structures inherent in gestural expressions. Further, the native reentrancy facilitates both the linking between modalities and the eventual situational grounding to contextual bindings.",

author = "Lucia Donatelli and Kenneth Lai and Richard Brutti and James Pustejovsky",

year = "2022",

doi = "10.1007/978-3-031-06018-2\_21",

language = "English",

isbn = "9783031060175",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "293--312",

editor = "Duffy, \{Vincent G.\}",

booktitle = "Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design",

note = "13th International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management, DHM 2022 Held as Part of the 24th HCI International Conference, HCII 2022 ; Conference date: 26-06-2022 Through 01-07-2022",

}

Donatelli, L, Lai, K, Brutti, R & Pustejovsky, J 2022, Towards Situated AMR: Creating a Corpus of Gesture AMR. in VG Duffy (ed.), Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part II. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13320 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 293-312, 13th International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management, DHM 2022 Held as Part of the 24th HCI International Conference, HCII 2022, Virtual, Online, 26/06/22. https://doi.org/10.1007/978-3-031-06018-2_21

Towards Situated AMR: Creating a Corpus of Gesture AMR. / Donatelli, Lucia; Lai, Kenneth; Brutti, Richard et al.
Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part II. ed. / Vincent G. Duffy. Springer Science and Business Media Deutschland GmbH, 2022. p. 293-312 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13320 LNCS).

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Towards Situated AMR

T2 - 13th International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management, DHM 2022 Held as Part of the 24th HCI International Conference, HCII 2022

AU - Donatelli, Lucia

AU - Lai, Kenneth

AU - Brutti, Richard

AU - Pustejovsky, James

PY - 2022

Y1 - 2022

N2 - In this paper, we extend Abstract Meaning Representation (AMR) in order to represent situated multimodal dialogue, with a focus on the modality of gesture. AMR is a general-purpose meaning representation that has become popular for its transparent structure, its ease of annotation and available corpora, and its overall expressiveness. While AMR was designed to represent meaning in language as text or speech, gesture accompanying speech conveys a number of novel communicative dimensions, including situational reference, spatial locations, manner, attitude, orientation, backchanneling, and others. In this paper, we explore how to combine multimodal elements into a single representation for alignment and grounded meaning, using gesture as a case study. As a platform for multimodal situated dialogue annotation, we believe that Gesture AMR has several attractive properties. It is adequately expressive at both utterance and dialogue levels, while easily accommodating the structures inherent in gestural expressions. Further, the native reentrancy facilitates both the linking between modalities and the eventual situational grounding to contextual bindings.

AB - In this paper, we extend Abstract Meaning Representation (AMR) in order to represent situated multimodal dialogue, with a focus on the modality of gesture. AMR is a general-purpose meaning representation that has become popular for its transparent structure, its ease of annotation and available corpora, and its overall expressiveness. While AMR was designed to represent meaning in language as text or speech, gesture accompanying speech conveys a number of novel communicative dimensions, including situational reference, spatial locations, manner, attitude, orientation, backchanneling, and others. In this paper, we explore how to combine multimodal elements into a single representation for alignment and grounded meaning, using gesture as a case study. As a platform for multimodal situated dialogue annotation, we believe that Gesture AMR has several attractive properties. It is adequately expressive at both utterance and dialogue levels, while easily accommodating the structures inherent in gestural expressions. Further, the native reentrancy facilitates both the linking between modalities and the eventual situational grounding to contextual bindings.

UR - https://www.scopus.com/pages/publications/85133023761

U2 - 10.1007/978-3-031-06018-2_21

DO - 10.1007/978-3-031-06018-2_21

M3 - Conference contribution

SN - 9783031060175

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 293

EP - 312

BT - Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design

A2 - Duffy, Vincent G.

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 26 June 2022 through 1 July 2022

ER -

Donatelli L, Lai K, Brutti R, Pustejovsky J. Towards Situated AMR: Creating a Corpus of Gesture AMR. In Duffy VG, editor, Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design: 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part II. Springer Science and Business Media Deutschland GmbH. 2022. p. 293-312. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2022 Jun 16. doi: 10.1007/978-3-031-06018-2_21

Towards Situated AMR: Creating a Corpus of Gesture AMR

Abstract

Publication series

Conference

Funding

UN SDGs

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this