CAIG: Foundation Models for Analyzing Images of Rock Cross-Sections using Procedural Synthetic Data

Rocks are the primary recorder of more than 99% of Earth’s History. Whether a rock chronicles processes deep in Earth’s interior, or on the wave agitated shores of a barrier island, rocks contain the clues that help geoscientists reconstruct ancient environments and piece together the history of life on Earth. Such study involves collecting hundreds or thousands of rock samples, photographing their cross-sections, sometimes reconstructing three-dimensional volumes, and ultimately creating a detailed catalogue of the contents of each rock, which can consist of hundreds or thousands of different objects and textures. For each embedded element, a geologist must identify its type (for example, fossil or crystal or bedform) and measure its shape, size, and abundance. Such cataloguing is traditionally done by hand and is very labor intensive, limiting scientific progress. This project seeks to develop new AI technologies that can automatically perform this cataloguing under the guidance of geoscientists, accelerating discovery and improving reproducibility. This research also is integrated with undergraduate and graduate education, and K12 outreach activities. This project aims to develop new AI capabilities to enable reliable automated analysis of images of rock cross-sections. The key idea is to address the bottleneck of training data, by creating a large amount of labeled synthetic data of rock cross-sections through computer graphics and physics-based simulation. In particular, the project employs a technique called “procedural generation", creating 3D objects and their simulations based on mathematical rules that allow precise control and unlimited variation. The synthetic data then are used to pretrain foundation models that can be further fine-tuned with a small number of real images to perform various visual analysis tasks on rock images. As a case study of advancing geoscience with the new AI capabilities, this project uses an extensive field collection from South Australia to test the hypothesis that the rise of metazoan reefs played a role in the coevolution of animals and their environments during the Cambrian radiation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.