Repository logo

Browsing by Author "Ortega, Francisco, committee member"

Now showing 1 - 12 of 12
  • Thumbnail Image
    ItemOpen Access
    Demonstrating that dataset domains are largely linearly separable in the feature space of common CNNs
    (Colorado State University. Libraries, 2020) Dragan, Matthew R., author; Beveridge, J. Ross, advisor; Ortega, Francisco, committee member; Peterson, Chris, committee member
    Deep convolutional neural networks (DCNNs) have achieved state of the art performance on a variety of tasks. These high-performing networks require large and diverse training datasets to facilitate generalization when extracting high-level features from low-level data. However, even with the availability of these diverse datasets, DCNNs are not prepared to handle all the data that could be thrown at them. One major challenges DCNNs face is the notion of forced choice. For example, a network trained for image classification is configured to choose from a predefined set of labels with the expectation that any new input image will contain an instance of one of the known objects. Given this expectation it is generally assumed that the network is trained for a particular domain, where domain is defined by the set of known object classes as well as more implicit assumptions that go along with any data collection. For example, some implicit characteristics of the ImageNet dataset domain are that most images are taken outdoors and the object of interest is roughly in the center of the frame. Thus the domain of the network is defined by the training data that is chosen. Which leads to the following key questions: Does a network know the domain it was trained for? and Can a network easily distinguish between in-domain and out-of-domain images? In this thesis it will be shown that for several widely used public datasets and commonly used neural networks, the answer to both questions is yes. The presence of a simple method of differentiating between in-domain and out-of-domain cases has significant implications for work on domain adaptation, transfer learning, and model generalization.
  • Thumbnail Image
    ItemOpen Access
    Digital twins for structural inspection, assessment, and management
    (Colorado State University. Libraries, 2023) Perry, Brandon J., author; Guo, Yanlin, advisor; Atadero, Rebecca, committee member; van de Lindt, John, committee member; Mahmoud, Hussam, committee member; Ortega, Francisco, committee member
    With the rapid advancements in remote sensing, uncrewed aircraft systems (UAS), computer vision, and machine learning, more techniques to maintain and evaluate the performance of the built infrastructure become available; however, these techniques are not always straightforward to adopt due to the remaining challenges in data analytics and the lack of executable actions that can be taken. The paper proposes a Digital Twin, which is a virtual representation of structures and has a myriad of applications to better assess and manage civil infrastructure. The proposed Digital Twin includes the techniques to store, visualize, and analyze the data collected from a UAS-enabled remote sensing inspection and computational models that support decision-making regarding the maintenance and operation of structures. The data analysis module identifies the location, extent, and growth of a defect over time, the structural components, and connections from the collected image with artificial intelligence (AI) and computer vision. In addition, the three-component (3C) dynamic displacements are measured from videos of the structure. A model library within the digital twin to assess the structure's performance, which includes three types of models, is proposed: 1) a visualization model to provide location-based data query, 2) an automatically generated finite element (FE) model as a basis for simulation, and 3) a surrogate model which can quickly predict a structure's behavior. Ultimately, the models in the library suggest executable actions that can be taken on a structure to better maintain and repair it. A discussion is presented showing how the Digital Twin can assist decision-making for structural management.
  • Thumbnail Image
    ItemOpen Access
    Does modality make a difference? A comparative study of mobile augmented reality for education and training
    (Colorado State University. Libraries, 2022) Kelley, Brendan, author; Humphrey, Michael, advisor; Martey, Rosa, committee member; Ortega, Francisco, committee member; Tornatzky, Cyane, committee member
    As augmented reality (AR) technologies progress they have begun to impact the field of education and training. Many prior studies have explored the potential benefits and challenges to integrating emerging technologies into educational practices. Both internal and external factors may impact the overall adoption of the technology, however there are key benefits identified for the schema building process, which is important for knowledge acquisition. This study aims to elaborate and expand upon prior studies to explore the question does mobile augmented reality provide for stronger knowledge retention compared to other training and education modalities? To address this question this study takes a comparative experimental approach by exposing participants to one of three training modalities (AR, paper manual, or online video) and evaluating their knowledge retention and other educational outcomes.
  • Thumbnail Image
    ItemOpen Access
    Embodied multimodal referring expressions generation
    (Colorado State University. Libraries, 2024) Alalyani, Nada H., author; Krishnaswamy, Nikhil, advisor; Ortega, Francisco, committee member; Blanchard, Nathaniel, committee member; Wang, Haonan, committee member
    Using both verbal and non-verbal modalities in generating definite descriptions of objects and locations is a critical human capability in collaborative interactions. Despite advancements in AI, embodied interactive virtual agents (IVAs) are not equipped to intelligently mix modalities to communicate their intents as humans do, which hamstrings naturalistic multimodal IVA. We introduce SCMRE, a situated corpus of multimodal referring expressions (MREs) intended for training generative AI systems in multimodal IVA, focusing on multimodal referring expressions. Our contributions include: 1) Developing an IVA platform that interprets human multimodal instructions and responds with language and gestures; 2) Providing 24 participants with 10 scenes, each involving ten equally-sized blocks randomly placed on a table. These interactions generated a dataset of 10,408 samples; 3) Analyzing SCMRE, revealing that the utilization of pointing significantly reduces the ambiguity of prompts and increases the efficiency of IVA's execution of humans' prompts; 4) Augmenting and synthesizing SCMRE, resulting in 22,159 samples to generate more data for model training; 5) Finetuning LLaMA 2-chat-13B for generating contextually-correct and situationally-fluent multimodal referring expressions; 6) Integrating the fine-tuned model into the IVA to evaluate the success of the generative model-enabled IVA in communication with humans; 7) Establishing the evaluation process which applies to both humans and IVAs and combines quantitative and qualitative metrics.
  • Thumbnail Image
    ItemEmbargo
    Functional methods in outlier detection and concurrent regression
    (Colorado State University. Libraries, 2024) Creutzinger, Michael L., author; Cooley, Daniel, advisor; Sharp, Julia L., advisor; Koslovsky, Matt, committee member; Liebl, Dominik, committee member; Ortega, Francisco, committee member
    Functional data are data collected on a curve, or surface, over a continuum. The growing presence of high-resolution data has greatly increased the popularity of using and developing methods in functional data analysis (FDA). Functional data may be defined differently from other data structures, but similar ideas apply for these types of data including data exploration, modeling and inference, and post-hoc analyses. The methods presented in this dissertation provide a statistical framework that allows a researcher to carry out an analysis of functional data from "start to finish''. Even with functional data, there is a need to identify outliers prior to conducting statistical analysis procedures. Existing functional data outlier detection methodology requires the use of a functional data depth measure, functional principal components, and/or an outlyingness measure like Stahel-Donoho. Although effective, these functional outlier detection methods may not be easily interpreted. In this dissertation, we propose two new functional outlier detection methods. The first method, Practical Outlier Detection (POD), makes use of ordinary summary statistics (e.g., minimum, maximum, mean, variance, etc). In the second method, we developed a Prediction Band Outlier Detection (PBOD) method that makes use of parametric, simultaneous, prediction bands that meet nominal coverage levels. The two new outlier detection methods were compared to three existing outlier detection methods: MS-Plot, Massive Unsupervised Outlier Detection, and Total Variation Depth. In the simulation results, POD performs as well, or better, than its counterparts in terms of specificity, sensitivity, accuracy, and precision. Similar results were found for PBOD, except for noticeably smaller values of specificity and accuracy than all other methods. Following data exploration and outlier detection, researchers often model their data. In FDA, functional linear regression uses a functional response Yi(t) and scalar and/or functional predictors, Xi(t). A functional concurrent regression model is estimated by regressing Yi on Xi pointwise at each sampling point, t. After estimating a regression model (functional or non-functional), it is common to estimate confidence and prediction intervals for parameter(s), including the conditional mean. A common way to obtain confidence/prediction intervals for simultaneous inference across the sampling domain is to use resampling methods (e.g., bootstrapping or permutation). We propose a new method for estimating parametric, simultaneous confidence and prediction bands for a functional concurrent regression model, without the use of resampling. The method uses Kac-Rice formulas for estimation of a critical value function, which is used with a functional pivot to acquire the simultaneous band. In the results, the proposed method meets nominal coverage levels for both confidence and prediction bands. The method we propose is also substantially faster to compute than methods that require resampling techniques. In linear regression, researchers may also assess if there are influential observations that may impact the estimates and results from the fitted model. Studentized difference in fits (DFFITS), studentized difference in regression coefficient estimates (DFBETAS), and/or Cook's Distance (D) can all be used to identify influential observations. For functional concurrent regression, these measures can be easily computed pointwise for each observation. However, the only current development is to use resampling techniques for estimating a null distribution of the average of each measure. Rather than using the average values and bootstrapping, we propose working with functional DFFITS (DFFITS(t)) directly. We show that if the functional errors are assumed to follow a Gaussian process, DFFITS(t) is distributed uniformly as a scaled Student's t process. Then, we propose using a multivariate Student's t distributional quantile for identifying influential functional observations with DFFITS(t). Our methodology ("Theoretical'') is compared against a competing method that uses a parametric bootstrapping technique ("Bootstrapped'') for estimating the null distribution of the mean absolute value of DFFITS(t). In the simulation and case study results, we find that the Theoretical method greatly reduces the computation time, without much loss in performance as measured by accuracy (ACC), precision (PPV), and Matthew's Correlation Coefficient (MCC), than the Bootstrapped method. Furthermore, the average sensitivity of the Theoretical method is higher in all scenarios than the Bootstrapped method.
  • Thumbnail Image
    ItemOpen Access
    Generalizations of comparability graphs
    (Colorado State University. Libraries, 2022) Xu, Zhisheng, author; McConnell, Ross, advisor; Ortega, Francisco, committee member; Cutler, Harvey, committee member; Hulpke, Alexander, committee member
    In rational decision-making models, transitivity of preferences is an important principle. In a transitive preference, one who prefers x to y and y to z must prefer x to z. Many preference relations, including total order, weak order, partial order, and semiorder, are transitive. As a preference which is transitive yet not all pairs of elements are comparable, partial orders have been studied extensively. In graph theory, a comparability graph is an undirected graph which connects all comparable elements in a partial order. A transitive orientation is an assignment of direction to every edge so that the resulting directed graph is transitive. A graph is transitive if there is such an assignment. Comparability graphs are a class of graphs where clique, coloring, and many other optimization problems are solved by polynomial algorithms. It also has close connections with other classes of graphs, such as interval graphs, permutation graphs, and perfect graphs. In this dissertation, we define new measures for transitivity to generalize comparability graphs. We introduce the concept of double threshold digraphs together with a parameter λ which we define as our degree of transitivity. We also define another measure of transitivity, β, as the longest directed path such that there is no edge from the first vertex to the last vertex. We present approximation algorithms and parameterized algorithms for optimization problems and demonstrate that they are efficient for "almost-transitive" preferences.
  • Thumbnail Image
    ItemOpen Access
    Helping humans and agents avoid undesirable consequences with models of intervention
    (Colorado State University. Libraries, 2021) Weerawardhana, Sachini Situmini, author; Whitley, Darrell, advisor; Ray, Indrajit, committee member; Pallickara, Sangmi, committee member; Ortega, Francisco, committee member; Seger, Carol, committee member
    When working in an unfamiliar online environment, it can be helpful to have an observer that can intervene and guide a user toward a desirable outcome while avoiding undesirable outcomes or frustration. The Intervention Problem is deciding when to intervene in order to help a user. The Intervention Problem is similar to, but distinct from, Plan Recognition because the observer must not only recognize the intended goals of a user but also when to intervene to help the user when necessary. In this dissertation, we formalize a family of intervention problems to address two sub-problems: (1) The Intervention Recognition Problem, and (2) The Intervention Recovery Problem. The Intervention Recognition Problem views the environment as a state transition system where an agent (or a human user), in order to achieve a desirable outcome, executes actions that change the environment from one state to the next. Some states in the environment are undesirable and the user does not have the ability to recognize them and the intervening agent wants to help the user in the environment avoid the undesirable state. In this dissertation, we model the environment as a classical planning problem and discuss three intervention models to address the Intervention Recognition Problem. The three models address different dimensions of the Intervention Recognition Problem, specifically the actors in the environment, information hidden from the intervening agent, type of observations and noise in the observations. The first model: Intervention by Recognizing Actions Enabling Multiple Undesirable Consequences, is motivated by a study where we observed how home computer users practice cyber-security and take action to unwittingly put their online safety at risk. The model is defined for an environment where three agents: the user, the attacker and the intervening agent are present. The intervening agent helps the user reach a desirable goal that is hidden from the intervening agent by recognizing critical actions that enable multiple undesirable consequences. We view the problem of recognizing critical actions as a multi-factor decision problem of three domain-independent metrics: certainty, timeliness and desirability. The three metrics simulate the trade-off between the safety and freedom of the observed agent when selecting critical actions to intervene. The second model: Intervention as Classical Planning, we model scenarios where the intervening agent observes a user and a competitor attempting to achieve different goals in the same environment. A key difference in this model compared to the first model is that the intervening agent is aware of the user's desirable goal and the undesirable state. The intervening agent exploits the classical planning representation of the environment and uses automated planning to project the possible outcomes in the environment exactly and approximately. To recognize when intervention is required, the observer analyzes the plan suffixes leading to the user's desirable goal and the undesirable state and learns the differences between the plans that achieve the desirable goal and plans that achieve the undesirable state using machine learning. Similar to the first model, learning the differences between the safe and unsafe plans allows the intervening agent to balance specific actions with those that are necessary for the user to allow some freedom. The third model: Human-aware Intervention, we assume that the user is a human solving a cognitively engaging planning task. When human users plan, unlike an automated planner, they do not have the ability to use heuristics to search for the best solution. They often make mistakes and spend time exploring the search space of the planning problem. The complication this adds to the Intervention Recognition Problem is that deciding to intervene by analyzing plan suffixes generated by an automated planner is no longer feasible. Using a cognitively engaging puzzle solving task (Rush Hour) we study how human users solve the puzzle as a planning task and develop the Human-aware Intervention model combining automated planning and machine learning. The intervening agent uses a domain specific feature set more appropriate for human behavior to decide in real time whether to intervene the human user. Our experiments using the benchmark planning domains and human subject studies show that the three intervention recognition models out performs existing plan recognition algorithms in predicting when intervention is required. Our solution to address the Intervention Recovery Problem goes beyond the typical preventative measures to help the human user recover from intervention. We propose the Interactive Human-aware Intervention where a human user solves a cognitively engaging planning task with the assistance of an agent that implements the Human-aware Intervention. The Interactive Human-aware Intervention is different from typical preventive measures where the agent executes actions to modify the domain such that the undesirable plan can not progress (e.g., block an action). Our approach interactively guides the human user toward the solution to the planning task by revealing information about the remaining planning task. We evaluate the Interactive Human-aware Intervention using both subjective and objective measures in a human subject study.
  • Thumbnail Image
    ItemOpen Access
    Multimodal agents for cooperative interaction
    (Colorado State University. Libraries, 2020) Strout, Joseph J., author; Beveridge, Ross, advisor; Ortega, Francisco, committee member; Daunhauer, Lisa, committee member
    Embodied virtual agents offer the potential to interact with a computer in a more natural manner, similar to how we interact with other people. To reach this potential requires multimodal interaction, including both speech and gesture. This project builds on earlier work at Colorado State University and Brandeis University on just such a multimodal system, referred to as Diana. I designed and developed a new software architecture to directly address some of the difficulties of the earlier system, particularly with regard to asynchronous communication, e.g., interrupting the agent after it has begun to act. Various other enhancements were made to the agent systems, including the model itself, as well as speech recognition, speech synthesis, motor control, and gaze control. Further refactoring and new code were developed to achieve software engineering goals that are not outwardly visible, but no less important: decoupling, testability, improved networking, and independence from a particular agent model. This work, combined with the effort of others in the lab, has produced a "version 2'' Diana system that is well positioned to serve the lab's research needs in the future. In addition, in order to pursue new research opportunities related to developmental and intervention science, a "Faelyn Fox'' agent was developed. This is a different model, with a simplified cognitive architecture, and a system for defining an experimental protocol (for example, a toy-sorting task) based on Unity's visual state machine editor. This version too lays a solid foundation for future research.
  • Thumbnail Image
    ItemOpen Access
    The influence of trust, self-confidence and task difficulty on automation use
    (Colorado State University. Libraries, 2023) Patton, Colleen E., author; Clegg, Benjamin, advisor; Wickens, Christopher, committee member; Fisher, Gwen, committee member; Ortega, Francisco, committee member
    Automation can be introduced statically or dynamically to help humans perform tasks. Static automation includes always-present automation types, whereas in dynamic automation, the presence of automation is controlled by another source, typically a human. In static automation, trust, automation accuracy, task difficulty and prior experience with the automation all contribute to the human dependence on the automation. In the dynamic literature however, a small body of research suggests that accuracy and task difficulty do not impact the decision to use automation, but a combination of trust and self-confidence does. The difference between the influence (or lack thereof) of task difficulty in static and dynamic automation is unusual, and prior literature does not make a strong case as to why this difference exists. Through three experiments, the influences of task difficulty, prior experience, trust, self-confidence, and their interactions are investigated. Experiment 1 used a dual task warehouse management paradigm with a lower-workload and higher-workload version of the task. Results indicated that trust-self-confidence difference was related to automation use, such that higher trust and lower self-confidence led to more use. Additionally, the difficulty manipulation did not have an impact on automation use, but self-confidence did not change across the two levels of difficulty. Experiment 2 investigated four levels of difficulty through a dynamic decision making task with participants detecting hostile ships. There was a difference in automation use at the easiest and most difficult levels, indicating that if the task difficulty difference is salient enough, it may influence automation use. The trust-self-confidence relationship was also present here, but these measures were only collected at the end of the task so their influence across the difficulty levels could not be measured. Experiment 3 used the same paradigm as Experiment 2 to investigate how perceived difficulty, as compared to objective difficulty, influences automation use. Results indicated that perceived workload influenced automation use, as did the change the trust-self-confidence difference. The findings of these experiments provide insight into how trust and self-confidence interact to influence the choice to use automation and provide novel evidence for the importance of workload in discretionary automation use decisions. This suggests the importance of consideration of human operator perceptions and beliefs about a system and of themselves when considering how often automation will be used. These findings create a foundation for a model of influences on automation use.
  • Thumbnail Image
    ItemOpen Access
    Throughput optimization techniques for heterogeneous architectures
    (Colorado State University. Libraries, 2024) Derumigny, Nicolas, author; Pouchet, Louis-Noël, advisor; Rastello, Fabrice, advisor; Hack, Sebastian, committee member; Rohou, Erven, committee member; Malaiya, Yashwant, committee member; Ortega, Francisco, committee member; Pétrot, Frédéric, committee member; Wilson, James, committee member; Zaks, Ayal, committee member
    Moore's Law has allowed during the past 40 years to exponentially increase transistor density of integrated circuits. As a result, computing devices ranging from general-purpose processors to dedicated accelerators have become more and more complex due to the specialization and the multiplication of their compute units. Therefore, both low-level program optimization (e.g. assembly-level programming and generation) and accelerator design must solve the issue of efficiently mapping the input program computations to the various chip capabilities. However, real-world chip blueprints are not openly accessible in practice, and their documentation is often incomplete. Given the diversity of CPUs available (Intel's / AMD's / Arm's microarchitectures), we tackle in this manuscript the problem of automatically inferring a performance model applicable to fine-grain throughput optimization of regular programs. Furthermore, when order of magnitude of performance gain over generic accelerators are needed, domain-specific accelerators must be considered; which raises the same question of the number of dedicated units as well as their functionality. To remedy this issue, we present two complementary approaches: on one hand, the study of single-application specialized accelerators with an emphasis on hardware reuse, and, on the other hand, the generation of semi-specialized designs suited for a user-defined set of applications.
  • Thumbnail Image
    ItemOpen Access
    Time sharing performance of egocentric and allocentric frames of reference as an indicator of resource pool
    (Colorado State University. Libraries, 2021) Patton, Colleen E., author; Clegg, Bemjamin, advisor; Wickens, Christopher, committee member; Ortega, Francisco, committee member
    The Multiple Resource Model (MRM) sets forth groups of cognitive resources and is used to predict dual task interference. Recent updates to the model suggest that it may not be all encompassing. The current studies aim to determine the resource use of egocentric and allocentric frames of reference (FoR) within the criteria of the MRM. Egocentric and allocentric FoR have been widely studied for their use in navigation aids, especially in aviation, and a plethora of neurological research has attempted to determine the neural correlates of each FoR. These two bodies of literature support the first two criteria of being considered separate resources, but the time sharing capabilities (the last criterion) have not been investigated. The current research used a dual task paradigm under intermediate and heavy resource use to determine how these FoR can be time shared. Results between experiments conflicted but indicated a stronger tendency toward improved performance under conditions in which the FoR being used for both tasks was the same. This was unexpected and does not fit into the MRM. Improved performance may be a result of task similarity, which can improve performance according to the shared processing routines hypothesis. Implications for navigation aid design are discussed.
  • Thumbnail Image
    ItemOpen Access
    Towards interactive analytics over voluminous spatiotemporal data using a distributed, in-memory framework
    (Colorado State University. Libraries, 2023) Mitra, Saptashwa, author; Pallickara, Sangmi Lee advisor; Pallickara, Shrideep, committee member; Ortega, Francisco, committee member; Li, Kaigang, committee member
    The proliferation of heterogeneous data sources, driven by advancements in sensor networks, simulations, and observational devices, has reached unprecedented levels. This surge in data generation and the demand for proper storage has been met with extensive research and development in distributed storage systems, facilitating the scalable housing of these voluminous datasets while enabling analytical processes. Nonetheless, the extraction of meaningful insights from these datasets, especially in the context of low-latency/ interactive analytics, poses a formidable challenge. This arises from the persistent gap between the processing capacity of distributed systems and their ever-expanding storage capabilities. Moreover, the interactive querying of these datasets is hindered by disk I/O, redundant network communications, recurrent hotspots, transient surges of user interest over limited geospatial regions, particularly in systems that concurrently serve multiple users. In environments where interactive querying is paramount, such as visualization systems, addressing these challenges becomes imperative. This dissertation delves into the intricacies of enabling interactive analytics over large-scale spatiotemporal datasets. My research efforts are centered around the conceptualization and implementation of a scalable storage, indexing, and caching framework tailored specifically for spatiotemporal data access. The research aims to create frameworks to facilitate fast query analytics over diverse data-types ranging from point, vector, and raster datasets. The frameworks implemented are characterized by its lightweight nature, residence primarily in memory, and their capacity to support model-driven extraction of insights from raw data or dynamic reconstruction of compressed/ partial in-memory data fragments with an acceptable level of accuracy. This approach effectively helps reduce the memory footprint of cached data objects and also mitigates the need for frequent client-server communications. Furthermore, we investigate the potential of leveraging various transfer learning techniques to improve the turn-around times of our memory-resident deep learning models, given the voluminous nature of our datasets, while maintaining good overall accuracy over its entire spatiotemporal domain. Additionally, our research explores the extraction of insights from high-dimensional datasets, such as satellite imagery, within this framework. The dissertation is also accompanied by empirical evaluations of our frameworks as well as the future directions and anticipated contributions in the domain of interactive analytics over large-scale spatiotemporal datasets, acknowledging the evolving landscape of data analytics where analytics frameworks increasingly rely on compute-intensive machine learning models.