Elements: A GPU-enabled exascale adaptive mesh code for multiphysics cosmological structure formation

This project develops an open-source community software package for simulating galaxy formation and evolution that runs efficiently on the largest supercomputers in the world, particularly supercomputers that are built using graphics processing units (GPUs). This approach enables researchers to run larger galaxy simulations, calculations that incorporate more physical processes, and simulations using larger numbers of galaxies than has previously been possible. This enables progress in understanding galaxy formation and evolution. In addition to creating a high performance, scalable tool for studying galaxies, this code is also usable in other fields of astrophysics such as star formation. The project uses modern, open-source software engineering practices that enable the software to be extended and improved upon through code contributions by other researchers. Finally, this project provides extensive documentation and other tools that allow it to be used by both new researchers as well as instructors of undergraduate and graduate astronomy and astrophysics courses, extending its scientific reach. Building on the Quokka Adaptive Mesh Refinement (AMR) radiation magnetohydrodnamics code, this project enables simulations of cosmological structure formation, with a focus on highly resolved, multiphysics galaxy evolution calculations. Quokka has much of the physics needed for this project, including self-gravitating magnetohydrodynamics, two-moment radiation transport, optically thin radiative cooling, and dark matter, star, and sink particles. This project extends Quokka to simulate cosmological volumes on GPUs with sufficient accuracy to resolve the internal structures of galaxies with all physics that is currently considered state-of-the-art. This project accomplishes this using grid-adaptive time-stepping, which, in combination with moving calculations from Central Processing Unit (CPU) systems to GPU-based systems, enables significant acceleration in time-to-solution compared to an equivalent CPU calculation. The project uses modern open-source scientific software development processes to make the code extensible and an inclusive approach to community-building to create a robust user-developer community that contributes to the long-term sustainability of the Quokka code. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.