CIF: Small: Learning Low-Dimensional Representations with Heteroscedastic Data Sources

As data-collection efforts continue to grow, so does heterogeneity in data. Machine-learning methods typically assume that data come from a single source or uniform instrumentation with noise characteristics that are the same for every data point. This project will address questions fundamental to learning low-dimensional data representations with heteroscedastic data, wherein samples from different sources have additive noise of different variances. It is well-known that classical linear dimensionality-reduction methods such as principal component analysis (PCA) are sensitive to outliers, so high-variance noise will degrade representations learned by PCA. However, robust methods that simply reject outliers are suboptimal if, indeed, the data do have some signal, even if it is buried in noise. The premise of this project therefore is to use approaches that learn the best way to incorporate the contribution of every different data source, no matter how high- or low-quality, to improve the overall learned representation. Many applications will benefit from the work, including medical imaging, environmental monitoring, astronomical data analysis, computer vision, and bioinformatics. The investigators' prior work in this area indicates that when learning is driven by heterogeneous and heteroscedastic sources – for example, in medical imaging, using data from multiple scanners, or with varying radiation levels – a better model will be learned by actively considering and modeling the heterogeneity. How to optimize learning in the face of such heterogeneity has been so far relatively unstudied, and this research aims to fill that gap. The technical contributions will be in three directions. First, the team of researchers will study open questions regarding how heterogeneity in data affects PCA, including establishing the required sample complexity for learning heteroscedastic models and assessing the optimization landscape of heteroscedastic PCA problems. Second, the team will extend heteroscedastic PCA methods and theory to consider union-of-subspaces models, dictionary learning models, and transform learning models. Third, the investigators will consider how nonlinear low-dimensional embedding methods are affected by heteroscedasticity in the data. The work will focus on distance-based methods and develop a foundational understanding of using distances in machine learning with heterogeneous data sources. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.