Repository logo

Browsing by Author "Krishnaswamy, Nikhil, author"

Now showing 1 - 2 of 2
  • Thumbnail Image
    ItemOpen Access
    A methodology for evaluating multimodal referring expression generation for embodied virtual agents
    (Colorado State University. Libraries, 2023-10-09) Alalyani, Nada, author; Krishnaswamy, Nikhil, author; ACM, publisher
    Robust use of definite descriptions in a situated space often involves recourse to both verbal and non-verbal modalities. For IVAs, virtual agents designed to interact with humans, the ability to both recognize and generate non-verbal and verbal behavior is a critical capability. To assess how well an IVA is able to deploy multimodal behaviors, including language, gesture, and facial expressions, we propose a methodology to evaluate the agent's capacity to generate object references in a situational context, using the domain of multimodal referring expressions as a use case. Our contributions include: 1) developing an embodied platform to collect human referring expressions while communicating with the IVA. 2) comparing human and machine-generated references in terms of evaluable properties using subjective and objective metrics. 3) reporting preliminary results from trials that aimed to check whether the agent can retrieve and disambiguate the object the human referred to, if the human has the ability to correct misunderstanding using language, deictic gesture, or both; and human ease of use while interacting with the agent.
  • Thumbnail Image
    ItemOpen Access
    Combating spatial disorientation in a dynamic self-stabilization task using AI assistants
    (Colorado State University. Libraries, 2024-11-24) Mannan, Sheikh Abdul, author; Hansen, Paige, author; Vimal, Vivekanand Pandey, author; Davies, Hannah N., auhtor; DiZio, Paul, author; Krishnaswamy, Nikhil, author; ACM, publisher
    Spatial disorientation is a leading cause of fatal aircraft accidents. This paper explores the potential of AI agents to aid pilots in maintaining balance and preventing unrecoverable losses of control by offering cues and corrective measures that ameliorate spatial disorientation. A multi-axis rotation system (MARS) was used to gather data from human subjects self-balancing in a spaceflight analog condition. We trained models over this data to create "digital twins" that exemplified performance characteristics of humans with different proficiency levels. We then trained various reinforcement learning and deep learning models to offer corrective cues if loss of control is predicted. Digital twins and assistant models then co-performed a virtual inverted pendulum (VIP) programmed with identical physics. From these simulations, we picked the 5 best-performing assistants based on task metrics such as crash frequency and mean distance from the direction of balance. These were used in a co-performance study with 20 new human subjects performing a version of the VIP task with degraded spatial information. We show that certain AI assistants were able to improve human performance and that reinforcement-learning based assistants were objectively more effective but rated as less trusted and preferable by humans.