CRII: HCC: RUI: Visualization-Based Multimodal Data Analysis for Qualitative Research

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). The goal of this project is to establish fundamental visual analysis strategies that integrate multimodal data and human-in-the-loop machine learning techniques to promote and support a transparent and trustworthy qualitative data analysis process. Qualitative researchers collect and analyze non-numerical data to understand people's interactions, opinions, and experiences. The presence of potential bias in qualitative research is a well-recognized problem, but research to increase transparency and trustworthiness in the analysis phases have been limited. This research develops a novel role for visualizations in addressing current challenges in qualitative data analysis through the integration of text analysis and multimodal data extraction. Given the prevalent use of qualitative research in academia, qualitative data analysis without transparency and verification can have far-reaching negative impacts such as discriminating policies, suboptimal patient-care, and reinforced stigmas. Thus, qualitative data analysis should be conducted in a rigorous manner to yield trustworthy and meaningful results. The techniques developed will significantly contribute to the data analysis, visualization, and human-computer interaction fields. Despite advances in qualitative data analysis software, there are three key dilemmas in the current qualitative research process: the cognitive burden resulting from the vast amount of data, subjectivity and potential bias that are introduced by the researcher, and the underutilization of multimodal data containing important non-verbal cues such as vocal tones and facial expressions. The specific objective of this proposal is to identify and demonstrate: (1) how text analysis techniques can be combined with visualization and human feedback to alleviate cognitive burden and lessen bias; and, (2) how visual summaries of audio features can promote the incorporation of non-verbal cues in qualitative analysis. An open-source visualization webtool supporting these enhanced analysis techniques will be developed using an iterative, user-centered design methodology. Prospective users with qualitative analysis experience will be selected for a participatory design session, usability tests, and a field study. The field study will be grounded in the case of sickle cell disease, a topic in which stigmatization of patients is common. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.