CAREER: Towards Disaggregated Persistent Key-Value Stores

Modern data centers are evolving to become more environmentally sustainable and efficient by adopting resource-disaggregated infrastructure, known as disaggregated data centers. This approach separates computing, storage, and networking resources to improve scalability, flexibility, and energy efficiency. Persistent key-value stores (PKVS)-which are essential for storage systems, databases, data deduplication, and big data analytics-are widely used in today's technology landscape. As cloud computing, big data, and large language models (LLMs) continue to grow rapidly, it is becoming crucial to adapt PKVS for use in disaggregated data centers. However, current PKVS face significant challenges in this environment, such as managing diverse hardware resources, scaling effectively, and handling risks like system failures, data corruption, and security threats. This CAREER proposal aims to create a flexible framework to redesign and optimize different types of PKVS for disaggregated data centers. The project will focus on better use of mixed hardware, scalable performance, and stronger reliability and security. Ultimately, this work will lead to greater resource and energy savings, extended hardware life, and better performance-to-cost balance for large-scale data systems. These improvements will benefit applications in cloud computing, machine learning, and scientific research. This project will devise new techniques to achieve key-range- and hotness-aware compute and data tiering with LLMs-assisted management, component-level scaling up and instance-level scaling out, and decentralized handling of execution failures and data protection with multi-level checksums and encryption. The proposed methodologies, system designs, and implemented components will benefit the memory and storage research communities in further developing storage systems and data infrastructures in disaggregated data centers. The project also tightly integrates research with education through various educational activities, including curriculum development and updates with the concept of "disaggregation," expanding winter and summer research camps to engage more students for cutting-edge research, especially undergraduates students, and enhancing interactions and collaborations with industry. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.