Collaborative Research: Exploring an Integrated Cognitive-Emotional State Assessment System to Support Real-Time Automation Intervention Design for Advanced Driving Systems

The primary objective of the research supported by this award is to inform real-time adaptive automation interventions in safety-critical system operations by leveraging cognitive workload predictions and integrating operator emotional state information. The research seeks to address two critical limitations in current cognitive workload modeling: (1) unreliable ground-truth labeling; and (2) classification model dependence on extensive offline training. By capturing multi-source physiological signals and context-dependent emotional state information, the project looks to formulate a new cognitive workload model to significantly enhance the accuracy of human state assessment. This work seeks to produce an integrated cognitive-emotional state assessment framework that links operator cognitive and emotional states to specific system automation responses. The research plan involves high-fidelity driving simulator experiments to generate a multimodal dataset that captures driver performance, central and peripheral physiological signals, cognitive workload responses, emotional states, and driver feedback during challenging scenarios. Statistical analyses intend to identify how cognitive and emotional states interact, providing a quantitative and probabilistic basis for decision rules to trigger timely, context-aware driver assistance. In parallel, nonlinear deep learning models will be trained to the simulator dataset to predict optimal timing and forms of automation interventions. The primary technical outcome intends to be a novel systems design framework for advanced driver assistance systems, achieving more accurate real-time state assessment and contextually appropriate automation interventions. The research looks to generate fundamental insights on the dynamic interplay of cognitive and emotional states in safety-critical tasks, representing a paradigm shift in how human-centered automation systems can enhance operator safety and trust. The project will also have an educational impact by training graduate students in human factors engineering and intelligent systems design. Finally, research findings will be disseminated broadly through peer-reviewed publications, conference presentations, and other outreach efforts to benefit both the research community and industry practitioners. This award has been funded by the Engineering Design and Systems Engineering, EDSE and Mind, Machine and Motor Nexus, M3X programs in the Division of Civil, Mechanical, and Manufacturing Innovation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.