Repository logo
 

A methodology for evaluating multimodal referring expression generation for embodied virtual agents

Simple item page
dc.contributor.authorAlalyani, Nada, author
dc.contributor.authorKrishnaswamy, Nikhil, author
dc.contributor.authorACM, publisher
dc.date.accessioned2024-11-11T19:34:33Z
dc.date.available2024-11-11T19:34:33Z
dc.date.issued2023-10-09
dc.description.abstractRobust use of definite descriptions in a situated space often involves recourse to both verbal and non-verbal modalities. For IVAs, virtual agents designed to interact with humans, the ability to both recognize and generate non-verbal and verbal behavior is a critical capability. To assess how well an IVA is able to deploy multimodal behaviors, including language, gesture, and facial expressions, we propose a methodology to evaluate the agent's capacity to generate object references in a situational context, using the domain of multimodal referring expressions as a use case. Our contributions include: 1) developing an embodied platform to collect human referring expressions while communicating with the IVA. 2) comparing human and machine-generated references in terms of evaluable properties using subjective and objective metrics. 3) reporting preliminary results from trials that aimed to check whether the agent can retrieve and disambiguate the object the human referred to, if the human has the ability to correct misunderstanding using language, deictic gesture, or both; and human ease of use while interacting with the agent.
dc.format.mediumborn digital
dc.format.mediumarticles
dc.identifier.bibliographicCitationNada Alalyani and Nikhil Krishnaswamy. 2023. A Methodology for Evaluating Multimodal Referring Expression Generation for Embodied Virtual Agents. In INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI '23 Companion), October 09–13, 2023, Paris, France. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3610661.3616548
dc.identifier.doihttps://doi.org/10.1145/3610661.3616548
dc.identifier.urihttps://hdl.handle.net/10217/239536
dc.languageEnglish
dc.language.isoeng
dc.publisherColorado State University. Libraries
dc.relation.ispartofPublications
dc.relation.ispartofACM DL Digital Library
dc.rights©Nada Alalyani, et al. ACM 2023. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ICMI '23, https://dx.doi.org/10.1145/3610661.3616548.
dc.subjectembodied agents
dc.subjectnon-verbal behaviours
dc.subjectmultimodality
dc.subjectreferring expression generation
dc.titleA methodology for evaluating multimodal referring expression generation for embodied virtual agents
dc.typeText

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
FACF_ACMOA_3610661.3616548.pdf
Size:
4.03 MB
Format:
Adobe Portable Document Format
Download

Collections

Publications