I-Corps: Translation Potential of Large Multimodal Language Models as Intelligent Assistants

This I-Corps project is based on the development of conversational artificial intelligence to create intelligent personal assistants. Current voice assistants are context-blind as they transcribe words while discarding the rich information that sound carries about a person's surroundings such as whether they are in a moving vehicle, a noisy restaurant, a clinic, or a quiet room, and who else is present and where. This technology closes this gap with assistants that perceive, interpret, and adapt to a user's full physical environment in real time. Applications include wearable earbuds, augmented-reality devices, automotive interfaces, hearing-health solutions, and workplace productivity tools. Use of this technology may lead to more natural consumer devices, safer hands-free interaction while driving, assistive listening, and tools that reduce information overload at work. This I-Corps project utilizes experiential learning coupled with first-hand investigation of the industry ecosystem to assess the translation potential of context-aware conversational artificial intelligence that is a new class of spatially grounded, full-duplex large audio-language models. Conventional speech systems treat audio as a single-channel, one-dimensional waveform stripped of spatial and environmental information, and operate in a half-duplex mode that forces rigid turn-taking. This technology advances three interconnected innovations. The first is a unified audio-native foundation model that performs joint reasoning across speech, music, environmental sound, and spatial audio with long, multi-turn context, rather than routing each sound type through a separate pipeline. The second is a spatially-aware speech-to-speech methods model with multi-microphone geometry, room acoustics, and directional source separation, enabling the system to localize speakers, resolve overlapping speech, and classify acoustic scenes in real time. The third is a full-duplex cognitive inference lets the system listen, reason, and speak simultaneously while calling external tools mid-utterance. Users may benefit from assistants that respond more naturally, suppress noise, and deliver contextually appropriate, privacy-preserving results on edge devices. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.