RI: Small: Characterizing the Meaning of Human Preferences for AI Alignment

Artificial Intelligence (AI) systems are becoming increasingly capable, yet deploying them to produce reliable, intended outcomes remains difficult. Large language models often fail to follow instructions, AI systems that govern important functions sometimes behave unpredictably, and autonomous systems, such as self-driving vehicles or robots, can act in ways that diverge from user expectations. To improve reliability and utility, future AI systems will need to demonstrate that their goals and behaviors consistently reflect and support the intentions of their human users. The dominant current paradigm for AI alignment relies on learning from human preferences over possible actions or outcomes of an AI system. However, such methods make a number of assumptions about how preferences should be interpreted and ignore many potential sources of error. The aim of this project is to improve the scientific characterization of human preferences in the context of AI alignment and leverage that knowledge to practically improve AI systems. More specifically, Reinforcement Learning from Human Feedback (RLHF) is now at the core of many of the most successful contemporary approaches to AI alignment in applications ranging from robotics to language modeling. RLHF aims to align a policy with the desires implied by human preferences between pairs of trajectories, outcomes, or model outputs. However, such approaches typically rely on very strong assumptions about the meaning of human preferences---for example, that a preference between two trajectories implies that one has a higher utility than another; or that non-utility-related factors do not add noise to preferences, such as the unequal cognitive effort required to evaluate each presented option. If the fundamental interpretation of preferences is flawed, then all of these approaches will also fall short. Thus, the goals of this proposed work are to: (1) Devise and perform human studies to systematically identify the drivers of human preferences in the context of AI alignment -- both core drivers and noise factors; (2) More accurately model core drivers of human preferences to improve the quality of RLHF and AI alignment; (3) Develop data collection best practices to eliminate or mitigate the effects of undesirable noise in human preferences; (4) Demonstrate that these techniques scale to complex problems of practical interest, including self-driving cars and large language models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.