I-Corps: Translation Potential of a Generative Artificial Intelligence (GenAI) Result Comparison Tool for Evaluating Foundation Model Outputs

This I-Corps project is based on the development of an evaluation framework for comparing results from Generative Artificial Intelligence (GenAI) systems. GenAI models are rapidly being deployed for complex data analyses in domains like health, finance, law and chemical manufacturing. Data analysis in high-stakes domains necessitates robust and reproducible evaluation methods. However, users today often create non-standardized scripts using their choice of metrics. This technology takes a domain-independent approach for evaluating model outputs across text, audio, and images. The approach can be specialized to industry domains to meet the growing demand for reliable, domain-aware evaluation tools as GenAI becomes increasingly embedded in areas such as aerospace, defense, healthcare, and industrial critical infrastructure. This may reduce user risks, improve accountability in AI deployment, and provide for responsible AI evaluation. This I-Corps project utilizes experiential learning coupled with first-hand investigation of the industry ecosystem to assess the translation potential of Generative Artificial Intelligence (GenAI) for high-impact application domains. GenAI evaluation methods treat all deviations from a reference as equivalent, which can hide domain-critical errors while overemphasizing trivial differences. This technology compares GenAI results while training Foundation Models (FMs) to specialized application domains. The initial focus is on health, and will use biosignals such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking (ET). This technology provides detailed evaluation of GenAI outputs in the realms of both reference-based, and reference-free visualizations, metrics, and insights. In addition, the system compares generated AI results in a domain-aware manner and produces severity-aware scores, visual explanations, and action reports. The approach integrates domain ontology, severity-weighted scoring, stage-level error attribution, escalation estimation, domain-specific visualization, and action dispatch in one evaluation workflow. The may enable the use FMs confidently, and to apply the framework to other areas. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.