TOWARDS SCIENCE-GUIDED KNOWLEDGE DISTILLATION FOR MULTIMODAL LEARNING UNDER CROSS-DOMAIN DISTRIBUTION SHIFT

Abstract

Modern multimodal artificial intelligence has demonstrated unprecedented generalization across broad domains. However, deploying these massive architectures in specialized scientific and op- erational settings, such as earth observation and precision agriculture, presents prohibitive computational and memory bottlenecks. Furthermore, these models often experience substantial performance degradation when confronted with extreme label sparsity and persistent distribution shifts arising from evolving sampling devices, diverse data modalities, and dynamic operational characteristics. This dissertation explores these challenges and presents a comprehensive methodology to bridge the adaptability gap by advancing the paradigm of Knowledge Distillation (KD). KD serves as a critical mechanism to extract broad intelligence from massive pre-trained foundation models, distilling it into lightweight, domain-specific student networks that respect strict computational limits. Building upon this concept, this work introduces three complementary frameworks designed to handle severe domain mismatches.First, we develop strategies for heterogeneous domain-shift mitigation that ensure representational consistency across varied spatiotemporal datasets. The proposed cross-resolution distillation framework distills coarse satellite observations into lightweight, locally precise models anchored by sparse in-situ measurements, demonstrating a 40% reduction in soil moisture prediction error. Second, we propose inverse knowledge distillation, which reverses the standard distillation paradigm by transferring architectural and representational knowledge from models trained on abundant, low-spectral-resolution observations to specialized models operating on high-spectral- resolution hyperspectral data. This approach achieves a 26% gain in reconstruction fidelity for 218- band imagery. Third, we establish science-guided domain adaptation through proposed physics-guided distillation frameworks, which embed domain-theoretic physical laws, specifically the Linear Spectral Mixing Model, directly into the neural architecture as differentiable constraints, improving structural fidelity by 23.6% and generalizing across geographically distinct regions. Collectively, these contributions enable the training of efficient neural networks that adapt to distribution shifts while remaining grounded in physical theory. Evaluations across diverse Earth observation datasets demonstrate substantial improvements in model robustness, computational efficiency, and representational fidelity over purely data-driven baselines. This research provides a principled path for deploying reliable, lightweight, and physically grounded AI systems in dynamic scientific domains.