CAREER: The Case for Disaggregated Database Systems

Database systems are fundamental to numerous mission-critical applications, such as finance, e-commerce, and transportation, because they efficiently manage large-scale data. For decades, database systems have been built for monolithic servers, where compute, memory, and storage are tightly integrated. However, these traditional databases now struggle to meet the stringent requirements of elasticity, scalability, and cost-effectiveness, especially when supporting large-scale applications in the cloud. Recently, there has been an emerging technology trend towards hardware resource disaggregation, which involves physically separating the hardware resources (such as compute, memory, and storage) into distinct resource pools to enable independent and elastic resource scaling. However, this disaggregated architecture presents fundamental challenges for traditional databases. This project will build a new database system that is specifically optimized for resource disaggregation to substantially improve performance, scalability, and elasticity. This will, in turn, significantly reduce costs for database customers and yield substantial economic benefits for society. The developed techniques will be open-sourced to enhance the research infrastructure. Research findings will be disseminated through publications at top-tier venues and will be incorporated into both graduate-level and undergraduate-level database courses at Purdue University. Furthermore, this project will actively engage students, including those who traditionally do not enroll in database systems. Specifically, this project investigates the profound impact of resource disaggregation (with respect to both storage and memory) on database systems. It presents a new disaggregated database system through three key thrusts. The first thrust introduces new approaches to efficiently manage database logs and perform transaction commits for storage disaggregation. The second thrust proposes new techniques to optimize database indexes and the buffer manager for memory disaggregation. The third thrust develops a new distributed database architecture for memory disaggregation and redesigns concurrency control and crash recovery. Overall, this project will drive the next wave of innovation in the field of database systems by starting a new line of research on disaggregated data(base) systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.