Collaborative Research: SHF: Medium: EPOD: Data Reduction for Science through Efficient Processing Without Decompression

Data reduction holds paramount importance in scientific endeavors and various data-intensive domains. This necessity is compounded by the unprecedented surge in data volume propelled by advancements in facilities and scientific research. Data compression stands as a prevalent method for data reduction. But prevailing solutions are hindered by a fundamental constraint: the requirement for decompression prior to processing. This introduces three primary challenges: (i) heightened strain on storage and memory resources, (ii) potentially lengthy decompression times for sizable datasets leading to significant workflow delays, and (iii) sometimes loss of accuracy when applications are compelled to operate on partial data views for space shortage. Aiming to develop a set of novel programming and runtime techniques, this project will create Efficient Processing without Decompression (EPOD), a novel approach to data reduction by lossless data compression while maintaining direct processability (without decompression) in the compressed state. The success of the project is poised to yield significant impacts, potentially reducing data sizes in numerous domains by one or two orders of magnitude without quality loss, while concurrently expediting data processing by multifold. The technique will help accelerate scientific research as well as improve the efficiency and productivity in various data-intensive domains. To develop the idea of EPOD into a new paradigm and solution for data reduction, this project includes five research activities: (i) developing the basic EPOD method and expanding its data coverage to floating-point datasets and so on; (ii) creating multi-level support of EPOD operations (i.e., data accesses or manipulations working directly on the EPOD compressed format) in forms of an EPOD library and a scalable large language model-based EPOD synthesizer; (iii) enabling continuous compression for streaming to make EPOD seamlessly integratable in streaming workflows of scientific facilities; (iv) developing advanced optimization of EPOD to maximize the efficiency and scalability; (v) integrating with existing data analytics ecosystem and complementary techniques and applications, and demonstrating the impact on a broad range of scientific domains and applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.