Data-driven selection of a convex loss function via shape-constrained estimation

This research project focuses on the notion of loss functions, which is central to machine learning and statistics. Loss functions measure the difference between the output predicted by the model and the actual output, and they typically satisfy a property called convexity so that they can be easily optimized. Loss functions quantify how accurate a model is at describing the data and therefore, almost all predictive models are computed by learning model parameters which minimize a given loss function. Choosing a good loss function is vitally important; a good loss function not only improves our predictions, but also allows us to build tighter confidence intervals, and gives us greater robustness to outliers. Although there are general guidelines for choosing a suitable loss function, these guidelines are qualitative and imprecise; most people still default to a few standard choices such as the square error loss. The goal of this project is to develop methods to estimate an optimal convex loss function from the data at hand. We will design, implement, and test algorithms that practitioners can use to automatically obtain loss functions specifically optimized to their dataset, which will allow the practitioners to make better predictive models. Successful execution of this project will have far-reaching effects on standard practices in data science. This project will be deeply integrated with the planned educational components at both the undergraduate and graduate levels. The first component of the project will look at linear regression and show that we can learn a data-driven convex loss function by approximating the unknown noise distribution with a log-concave density in a distributional distance known as the Fisher divergence. The proposed approach is computationally simple and, in settings where the noise is non-Gaussian, significantly improves upon the traditional squared error loss in estimation accuracy, inference quality, and robustness. The second component of the project will extend the idea to the setting of multi-task regression where the response is multivariate. The third component of the project will analyze the theoretical properties of score matching–the statistical method that underpins the first two components on convex loss estimation as well as being of fundamental importance in various other applications in statistical learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.