Narayana Rao Gari, Pradyumna Kumar, authorDraper, Bruce A., advisorBeveridge, Ross, committee memberPeterson, Chris, committee member2007-01-032007-01-032014http://hdl.handle.net/10217/88582Activity recognition in Computer Vision involves recognizing the appearance of an object of interest along with its action, and its relation to the scene or other important objects. There exist many methods that give this information about an object. However, these methods are noisy and are independent of each other. So, the mutual information between the labels is lost. For example, an object might be predicted to be a tree, whereas its action might be predicted as walk. But, trees can't walk. However, the compositional structure of the events is reflected by the compositional structure of natural language. The object of interest is the predicate, usually a noun, the action is the verb, and its relation to the scene may be a preposition or adverb. The lost mutual information that says that trees can't walk is present in natural language. The contribution of this thesis is a method of visual information fusion based on exploiting the mutual information from Natural language databases. Although Hidden Markov Models (HMM) are the traditional way to smooth noisy stream of data by integrating information across time, they can't account for the lost mutual information. This thesis proposes an extension to HMM (Consistent HMM) that can integrate visual information to the lost mutual information by exploiting the knowledge from language databases. Consistent HMM performs better than other state of the art HMMs on synthetic data generated to simulate the real world behavior. Although the performance gain of integrating the knowledge from language databases both during training phase and run-time is better, when considered individually, the performance gain is more when the knowledge is integrated during run-time than training.born digitalmasters thesesengCopyright and other restrictions may apply. User is responsible for compliance with all applicable laws. For information about copyright law, please see https://libguides.colostate.edu/copyright.computer visionhidden Markov modelsinteracting hidden Markov modelsnatural languageontologyvideo activity recognitionText