CAREER: Heterogeneous Neuromorphic and Edge Computing Systems for Realtime Machine Learning Technologies

Machine learning systems in robotics, self-driving cars, assistive technologies, and Internet-of-Things (IoT) applications require low-energy, real-time computation. Lower energy use ensures extended battery life for these battery-powered devices. This project focuses on American sign language translation to showcase its societal impact. To create practical sign language translation technology, multiple computer vision and language models are essential for seamless communication between sign language users and others. The aim is to deploy this on portable, wearable devices for on-demand use - a complex challenge. The research team will investigate breaking down these complex systems, distributing computation across interconnected tiny devices specialized in specific tasks. Beyond sign language translation, the methodology and framework developed in this project can pave the way for real-time technology in social robotics and smart manufacturing, among other domains. This project involves various educational and outreach initiatives, including developing cross-disciplinary curricula, generating online educational resources, engaging both undergraduate and high school students in research, and collaborating with industry partners to promote social robotics for K-5 learning. This project aims to harness the combined capabilities of neuromorphic and edge computing to forge a heterogeneous machine learning system. Its primary goal is to enable computer vision and language models on resource- and energy-constrained devices at an unprecedented scale. It focuses on several key aspects: (1) developing hybrid models that merge the energy efficiency, temporal sparsity, and spatiotemporal processing of spiking neural networks with the global processing of transformer models for complex large-scale computer vision tasks, (2) creating a methodology to deploy large language models on edge devices by employing system-level innovations such as computational graph modifications, custom kernels, and mathematical refactoring, (3) designing a flexible edge artificial intelligence (AI) accelerator to overcome hardware limitations hindering real-time implementation of large transformer models at the edge, (4) seamlessly integrating a heterogeneous system of mobile processors, edge AI accelerators, and neuromorphic hardware for a comprehensive end-to-end solution. Throughout the project, rigorous investigation delves into critical trade-offs between bandwidth, accuracy, performance, and energy consumption. This project is jointly funded by the Software and Hardware Foundation (SHF) core research program and the Established Program to Stimulate Competitive Research (EPSCoR). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.