CNS: Medium: Scaling the bandwidth-per-TB wall with declarative distributed storage interfaces

Large distributed storage systems within datacenters are primary components of cloud, Internet service, and data analytics infrastructures, and storage capacity demand is growing rapidly with the rise of data science, machine learning, and artificial intelligence. Analysts estimate that 175 zettabytes of data will be generated annually by 2025, most of which will be stored in data centers, which would require over 8 billion 20terabyte (20TB) mechanical Hard Disk Drives (HDDs) before accounting for data redundancy needed to protect data from device failures. Since such numbers are untenable, new technologies that allow higher capacity devices are being created, but they do not provide greater performance. The result is a coming performance “wall”, where the available access bandwidth-per-TB of stored data is too low to allow that data to be stored, maintained, and used. The goal of this research is to reimagine the decades-old application interfaces used for datacenter storage to enable large reductions in the bandwidth needed so that higher-capacity devices can be used. Existing input/output (IO) interfaces are imperative, such as “read this now” or “put this now”, which is easy for programmers but restrictive and inefficient for the system. This research will develop new “declarative” IO interfaces and orchestration approaches that allow programmers to express larger data access plans/needs and thereby allow the storage system to coordinate, coalesce, and schedule IO to minimize the aggregate bandwidth-per-TB needed by the various applications and data maintenance tasks required for reliable datacenter storage. The results will be more sustainable and cost-effective datacenter storage, eliminating the need to manufacture and deploy millions of HDDs, reducing Flash and DRAM cache requirements, and enabling deployment of new data maintenance activities that make data more useful, secure, and reliable. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.