CAREER: Robust Learning from Preference Feedback

In recent years, preference feedback—comparative inputs such as “A is better than B”—has emerged as a vital resource for guiding decision-making systems. Unlike explicit labels, preference feedback is often easier to collect and can be particularly valuable in subjective tasks where defining ideal outcomes is difficult. However, real-world preference data are often noisy, sparse, and heterogeneous, posing significant challenges to existing statistical methods. For example, recommendation systems may encounter incomplete feedback from users who abandon tasks due to fatigue or provide inconsistent inputs due to individual biases. This project aims to address the challenges of learning from preference feedback by developing robust statistical methods and advancing the theoretical foundations of preference-based learning. Additionally, it seeks to prepare students to tackle these challenges by integrating the research findings into innovative teaching platforms and educational curricula. The project will focus on three key areas of learning from preference feedback: ranking from pairwise comparisons, user-item rating systems, and reinforcement learning from human feedback. To advance the field, the project will (1) develop robust algorithms for ranking that account for ill-conditioned sampling mechanisms and relax parametric modeling assumptions; (2) propose new estimation and uncertainty quantification methods for user-item ratings that work effectively in sparse and heterogeneous settings; and (3) introduce novel frameworks for reinforcement learning that incorporate “out-of-list” preference feedback while addressing the issue of distribution shifts. Through these contributions, the project will bridge the gap between statistical theory and practical applications, creating tools to enhance decision-making systems across diverse domains. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.