Collaborative Research: Small: Immersive Multimodal Plan-to-Grasp Neural Decoders: Integrating Deep Graph-based and Adaptive Shared Control Theories

This project seeks to advance the science of brain-computer interfaces (BCIs), in the context of helping people with upper limb impairments use brain activity to control prosthetic arms to reach and grasp objects. Current control techniques have real limitations: they generally either require surgically implanted probes to capture brain activity or support a limited range of commands. To reduce these limitations, the project team will advance non-invasive methods for understanding and decoding "plan-to-grasp" signals in the brain that coordinate vision, movement planning, and hand control when reaching for objects. By combining these decoded plan-to-grasp signals with methods for shared human and AI control of robots, the project will develop new ways to control neuroprosthetic arms with a wider range of motion and abilities than existing methods. The project will also support education and workforce development through student training, K-12 outreach activities, sharing open-source data and software, and collaborations with community organizations focused on independent living and rehabilitation in Rhode Island. The project is structured around three main research activities. The first is to develop an immersive multimodal BCI framework that simultaneously records non-invasive brain activity, virtual video, and kinematic data during natural plan-to-grasp tasks in virtual reality. The second is to create large graph-based deep learning models to decode complex neural dynamics, classify grip types, and predict movement kinematics. The third is to implement a personalized real-time closed-loop virtual assistive arm-hand system and validate its performance on users with upper-limb impairment. Together, the work will lead to fundamental advances in neuroscience, advancing understanding of macroscale neural representations and dynamic cortical interaction mechanisms underlying active observation of the natural plan-to-grasp task using large graph-based nonlinear models and multidimensional spatial, temporal, and spectral features. There will also be significant advances around BCI-based prosthetic control, via combining these neurological signals insights with adaptive, personalized, and closed-loop shared control theories to support complex plan-to-grasp tasks. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.