EAGER: Language as Interface: Toward Integrated Multimodal Architectures for Grounded Intelligence

Large language models have changed how artificial intelligence is understood because success in language is easily mistaken for success in general intelligence. Earlier advances in computer vision were interpreted as progress in machine perception: systems became better at recognizing, detecting, segmenting, and tracking visual information. By contrast, fluent language output creates the impression that a system can reason, explain, plan, imagine, and reflect, because these forms of intelligence become visible to people through language. This project explores a new research paradigm in which language is treated not as the foundation of intelligence, but as an interface to other forms of intelligence, specifically, perception and reasoning. The project is high risk because it tests an early-stage architectural hypothesis rather than improving existing systems through standard methods such as larger datasets, fine-tuning, or prompting. It is potentially transformative because it may establish a new architectural direction for multimodal artificial intelligence: systems in which language does not merely describe or explain outputs but is explicitly coupled with structured presentations of perception and reasoning. This work can contribute to building robust, reliable and transparent artificial intelligence systems for education, science, medicine, robotics, and human-computer interaction, while training students in interdisciplinary methods that connect language, vision, cognitive science, and artificial intelligence evaluation. This project investigates the hypothesis that language can serve as an effective interface to intelligence only when linguistic expressions are connected to structured, inspectable, and updateable non-linguistic representations. The research will define a minimal language-facing representation that includes objects, attributes, relations, spatial structure, grounded predicates, uncertainty estimates, and update operations. It will then develop and evaluate three prototype interface designs: a) an object-centered interface in which referring expressions bind to persistent object representations, b) a predicate-grounding interface in which terms such as "behind," "blocking," "supporting," and "more than" map to perceptual or reasoning operations, and c) a bidirectional constraint interface in which a structured scene states guide language generation and explanation. These prototypes will be compared with standard multimodal prompting and tool-calling approaches. Evaluation will measure answer accuracy, reference stability, predicate grounding, spatial reasoning, and explanation faithfulness. If successful, the project will provide preliminary evidence and design principles for artificial intelligence systems in which language does not merely verbalize plausible outputs, but interfaces with structured perception and reasoning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.