Collaborative Research: AF: Small: A Unified Framework for Analyzing Adaptive Stochastic Optimization Methods Based on Probabilistic Oracles

Data science and machine learning have transformed modern science, engineering, and business. One of the pillars of modern-day machine-learning technology is mathematical optimization, which is the methodology that drives the process of learning from available and/or real-time generated data. Unfortunately, however, despite the successes of certain optimization techniques, large-scale learning remains extremely expensive in terms of time and energy, which puts the ability to train machines to perform certain fundamental tasks exclusively in the hands of those with access to extreme-scale supercomputing facilities. A significant deficiency of many contemporary techniques is that they "launch" an algorithm with a prescribed "trajectory," despite the fact that the actual trajectory that the algorithm will follow depends on unknown factors. Contemporary optimization techniques for machine learning essentially account for this by "tuning" algorithmic parameters, which means that the target is typically only hit after numerous expensive misses. Another significant deficiency of contemporary techniques is the restrictive set of assumptions often made about the optimization being performed, which typically includes the assumption that the machine-learning model is being trained with uncorrupted data. Modern real-world applications are far more complex. This project will explore the design and analysis of adaptive ("self-tuning") optimization techniques for machine learning and related topics. One goal is to produce adaptive algorithms with rigorous guarantees that can avoid the extreme amounts of wasteful computation that are required by contemporary algorithms for parameter tuning. Another goal is to extend the use of these algorithms to settings with imperfect data/information, which may be due to biased function information, corrupted data, or novel techniques for approximating the objective. Finally, many applications ultimately require the learning process or model to satisfy some explicit or implicit constraints. Optimization methods for such machine-learning applications are still in their infancy, largely due to their more complicated nature and further dependence on algorithmic parameters. This project aims to design a unified framework for analyzing adaptive stochastic optimization methods that will offer researchers and practitioners a set of easy-to-use tools for designing next-generation algorithms for cutting-edge applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.