Practical aspects of designing and developing a multimodal embodied agent
| dc.contributor.author | Bangar, Rahul, author | |
| dc.contributor.author | Beveridge, Ross, advisor | |
| dc.contributor.author | Ortega, Francisco R., advisor | |
| dc.contributor.author | Peterson, Christopher, committee member | |
| dc.date.accessioned | 2021-06-07T10:19:36Z | |
| dc.date.available | 2021-06-07T10:19:36Z | |
| dc.date.issued | 2021 | |
| dc.description.abstract | This thesis reviews key elements that went into the design and construction of the CSU CwC Embodied agent, also known as the Diana System. The Diana System has been developed over five years by a joint team of researchers at three institutions – Colorado State University, Brandeis University and the University of Florida. Over that time, I contributed to this overall effort and in this thesis, I present a practical review of key elements involved in designing and constructing the system. Particular attention is paid to Diana's multimodal capabilities that engage asynchronously and concurrently to support realistic interactions with the user. Diana can communicate in visual as well as auditory modalities. She can understand a variety of hand gestures for object manipulation, deixis, etc. and can gesture in return. Diana can also hold a conversation with the user in spoken and/or written English. Gestures and speech are often at play simultaneously, supplementing and complementing each other. Diana conveys her attention through several non-verbal cues like slower blinking when inattentive, keeping her gaze on the subject of her attention, etc. Finally, her ability to express emotions with facial expressions adds another crucial human element to any user interaction with the system. Central to Diana's capabilities is a blackboard architecture coordinating a hierarchy of modular components, each controlling a part of Diana's perceptual, cognitive, and motor abilities. The modular design facilitates contributions from multiple disciplines, namely VoxSim/VoxML with Text-to-speech/Automatic Speech Recognition systems for natural language understanding, deep neural networks for gesture recognition, 3D computer animation systems, etc. – all integrated within the Unity game engine to create an embodied, intelligent agent that is Diana. The primary contribution of this thesis is to provide a detailed explanation of Diana's internal working along with a thorough background of the research that supports these technologies. | |
| dc.format.medium | born digital | |
| dc.format.medium | masters theses | |
| dc.identifier | Bangar_colostate_0053N_16427.pdf | |
| dc.identifier.uri | https://hdl.handle.net/10217/232474 | |
| dc.language | English | |
| dc.language.iso | eng | |
| dc.publisher | Colorado State University. Libraries | |
| dc.relation.ispartof | 2020- | |
| dc.rights | Copyright and other restrictions may apply. User is responsible for compliance with all applicable laws. For information about copyright law, please see https://libguides.colostate.edu/copyright. | |
| dc.subject | embodied agent | |
| dc.subject | human computer interfaces | |
| dc.subject | deep learning | |
| dc.subject | multimodal systems | |
| dc.subject | gesture recognition | |
| dc.title | Practical aspects of designing and developing a multimodal embodied agent | |
| dc.type | Text | |
| dcterms.rights.dpla | This Item is protected by copyright and/or related rights (https://rightsstatements.org/vocab/InC/1.0/). You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s). | |
| thesis.degree.discipline | Computer Science | |
| thesis.degree.grantor | Colorado State University | |
| thesis.degree.level | Masters | |
| thesis.degree.name | Master of Science (M.S.) |
Files
Original bundle
1 - 1 of 1
Loading...
- Name:
- Bangar_colostate_0053N_16427.pdf
- Size:
- 2.57 MB
- Format:
- Adobe Portable Document Format