Collaborative Research: CAIG: Taming Chaos to Learn Atmospheric Physics

Efforts to protect life and property from weather hazards rely on computer models of the atmosphere to make weather forecasts and simulate long-term changes in the frequency and severity of weather extremes. The success of these models depends on their ability to capture the physics of the atmosphere, which in turn requires the calibration of dozens of adjustable settings that control the behavior of small-scale processes such as cloud formation. Choosing these settings well is essential for trustworthy predictions and projections but doing so is currently a slow, expensive, and partly manual process. This project develops new mathematical and computational methods to calibrate atmospheric models by taking advantage of a new, fast model that can automatically report how its outputs change when its internal settings change. Combining this capability with machine learning, the project aims to make atmospheric models more accurate, more stable, and easier to interpret, thus increasing their value for addressing national needs. The resulting methods have applications to other fields involving chaotic nonlinear systems, including plasma physics, turbulence, and geophysics. In addition, the project will train graduate students at the interface of artificial intelligence and Earth science, building a workforce fluent in both domains. The project leverages a new differentiable atmospheric general circulation model, which supports automatic differentiation and runs efficiently on modern hardware. The central hypothesis is that model gradients can inform calibration even over long timescales relevant to atmospheric statistics. The work is organized around three questions: over what spatiotemporal scales atmospheric gradients are informative; how to optimize a noisy, nonlinear hybrid model that combines physical and machine-learned components; and what constitutes the optimal representation of sub-grid atmospheric physics. The team develops a hybrid optimization framework that pairs gradient-based methods for short-term trajectory matching with ensemble Kalman inversion for long-term statistical matching, incorporating nudging to remove dependence on initial and boundary conditions. Additional activities include interpretable neural network parameterizations trained against high-resolution simulations, online learning via causation-entropy boosting to handle regime shifts, Pareto-based analysis of the accuracy-complexity tradeoff, and differentiable Bayesian optimal experimental design for active data acquisition. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.