Collaborative Research: Elements: A task-based code for multiphysics problems in astrophysics at exascale

Observing and modeling the merger of neutron stars with each other and with black holes are challenging tasks that push understanding of the universe as embodied by Einstein’s equations for general relativity. Observatories such as National Science Foundation-funded Laser Interferometer Gravitational-Wave Observatory (LIGO) need the results of computational models to help understand the signals they receive. Computational modeling of neutron star mergers in turn is extremely demanding and requires the use of the most advanced supercomputers. Even on these unique computing resources operated by the US Department of Energy and the National Science Foundation, computer models may still require weeks to run. Improvements to gravitational wave modeling software that take advantage of improved algorithms have the potential to reduce this execution time down to hours. This project improves the open-source gravitational wave-modeling software SpECTRE to use new algorithms and to make optimal use of one-of-a-kind supercomputing resources. The results from these computations are needed for scientists to understand black holes, the expansion of the universe, and how stars explode and leave behind black holes. The transformative techniques used by SpECTRE have the potential to also be applied to research areas in fluid dynamics, geoscience, plasma physics, and nuclear physics and engineering. The project is training the next generation of computational astrophysicists on the use and extension of SpECTRE through summer schools. The investigating team engages the public through visualizations and movies posted on social media and through public outreach events. The new SpECTRE code uses a hybrid finite difference-discontinuous Galerkin method, task-based parallelism, and the U.S. cyberinfrastructure Graphical Processing Unit (GPU) abstraction library Kokkos to accomplish its goals. This framework will allow multiphysics applications to be treated both accurately and efficiently on the new architectures of petascale and exascale machines. The code is designed to scale to over a million cores for efficient exploration of the parameter space of potential sources and allowed physics, and for the high-fidelity predictions needed to realize the promise of multi-messenger astronomy. The software design separates parallelism and physics capabilities in a way that makes adopting new computing paradigms and libraries possible without rewriting the physics modules. The code will allow astrophysicists to understand electromagnetic transients and gravitational-wave phenomena in compact objects, to reveal the dense matter equation of state, and to perform binary black hole simulations at the accuracies necessary for next-generation detectors. The key algorithmic innovations in the code, the hybrid finite difference-discontinuous Galerkin method coupled with task-based parallelism and GPU offloading, promise revolutionary impact in other fields relying on numerical solution of partial differential equations at the exascale. This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Division of Physics and the Division of Astronomy in the Mathematical and Physical Sciences Directorate. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.