SHF: Small: Efficient Multi-Task Learning for Augmented Reality Systems

Augmented Reality (AR) and Machine Learning (ML) are rapidly evolving fields that have potential to transform numerous industries by enabling immersive and intelligent applications. However, embedding advanced ML capabilities into AR devices is challenging because of limited hardware resources and the need to process large volumes of real-time sensor data. This award addresses these challenges by designing resource-efficient techniques that reduce computational load and energy consumption while maintaining high accuracy. The outcomes of this work have the potential to benefit a wide range of domains, including healthcare, education, and entertainment, by increasing the accessibility and reliability of AR technologies. In addition, the project includes a comprehensive education and outreach plan, which includes providing research experiences for undergraduate students, developing new computer engineering courses, engaging with high school students, and facilitating technology transfer to industry. This project focuses on a multi-task learning framework that integrates transformer- and convolution-based architectures with low-rank decomposition to enable efficient fine-tuning on resource-constrained AR devices. Task-aware dynamic feature sharing is employed to adaptively allocate computational resources, and quantization strategies are explored to balance performance and resilience against adversarial attacks. An adaptive policy network for inference is developed to accommodate real-time decision-making, and task scheduling algorithms are designed to optimize throughput across heterogeneous processing units. By evaluating these methods on real AR devices using diverse benchmarks, the project establishes foundational strategies for effectively deploying ML in AR systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.