Collaborative Research: CISE MSI: RCBP: RI: Explainable Multi-modality Time Series Question Answering System

Large Language Model (LLM) has become an emerging tool for complex reasoning tasks in data modalities such as image and video, by efficiently harnessing rich unlabeled data. Yet, few researcher have focused on time series data, which is widely used in critical applications with limited amount of annotation. Time series data poses three key unresolved challenges: 1) there is a lack of high-quality text information for help with identification that aligns with time series data; 2) deep learning models to reason with both time series and text data are under-researched; and 3) novel explainable artificial intelligence (AI) tools to ensure the trustworthiness of such models and give users confidence in model predictions are lacking. This project aims to address these challenges and develop a time series text-based cross-modality Question Answering (QA) system. The project will also promote close collaboration between UTRGV and Yale University to encourage Hispanic undergraduate students to pursue higher education and build UTRGV's capacity to conduct research on advanced AI topics including LLM, time series, foundational model, and trustworthy AI at UTRGV. This project aims to develop a time series text-based QA system that can accelerate a wide range of research by providing expert-level explanations. The technical aims of this research project can be divided into three key components: 1) Develop an automated high-quality time series annotation pipeline by designing a multi-view prompt-based QA generation framework to build training data and labels. 2) Develop a novel text and time series cross-modality pre-trained model to better enable knowledge extraction from time series data and fuse information across two modalities. 3) Enhance transparency and interpretability of the built model by developing a combination of time series-oriented explanation and text-oriented explanation. Collaborating with Idaho National Lab, the system will be evaluated by analyzing and forecasting extreme weather that can cause energy infrastructure damage, using the associated time series and text information. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.