CRII: CSR: Designing a Memory-Adaptive Query Execution Framework for Optimizing Resource Utilization and Cost Efficiency in Cloud Databases

Cloud-based data systems are central to modern applications, but they often rely on static resource allocation, such as reserving memory and compute power based on peak needs. This leads to underutilized resources, high costs, and inefficient performance. While cloud platforms offer scalability, most database systems cannot adjust resource usage during query execution. As workloads become more dynamic and data volumes grow, improving resource adaptability is essential. This project explores real-time, query-level resource adaptation to improve efficiency and reduce costs, helping to enhance national economic competitiveness. The project also creates educational and research opportunities for students through hands-on open-source development and mentorship programs. In this project, the researchers implement DynEx, a fully dynamic, memory-adaptive query execution engine for cloud database management systems (DBMSs) that aims to improve resource efficiency, cost-effectiveness, and performance. The research addresses three core questions: (1) how to design join and sort operators that adapt to memory fluctuations without overspilling or degrading performance; (2) how to coordinate a performance-aware resource broker and scheduler to reallocate memory based on query priorities and runtime demands; and (3) how to trigger auto-scaling to maintain performance targets while reducing cost and delay. DynEx builds on a shared-process architecture with disaggregated storage, integrated into the open-source AsterixDB system. The engine performance benchmarks include dynamic workloads using synthetic and real datasets to evaluate adaptability, throughput, and resource use. The release of all resulting code and artifact helps support reproducibility and future research. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.