OAC Core: A Scalable and Deployable Container Orchestration Cyber Infrastructure Toolkit for Deploying Big Data Analytics Applications in Public Cloud

Today many big data analytics applications (e.g., fraud detection, social data analytics, education, climate modeling, epidemiology, and finance) need to process enormous datasets from geographically distributed locations. An emerging trend is to host these big data analytics applications in the public cloud. They can be packaged to run in a lightweight isolated execution environment (containers) and deployed on computing resources rented from public cloud providers, which can be updated and scaled seamlessly. However, the complex inter-container correlations and the heterogeneity of hardware resources pose significant challenges in managing these big data analytics applications in the public cloud. This project enables the easy deployment of containerized big data analytics applications in the public cloud and provides cloud providers with insights to better tune their systems for current and future big data workloads. The goal of this project is to develop a scalable and deployable cyber infrastructure (CI) container orchestration toolkit for deploying large numbers of containerized big data analytics applications on heterogeneous nodes in state-of-the-art public multi/hybrid-cloud. This project spans three complementary thrusts: (i) a novel “black-box” lightweight tool is implemented, which detects inter-container correlations for containerized big data analytics applications in a non-intrusive manner via hierarchical clustering and co-occurrence analysis; (ii) a novel scalable container scheduler is implemented, which deploys containerized big data analytics applications on heterogeneous nodes in the public cloud in a correlation-aware manner; and (iii) the system is implemented on open-source container orchestration tools and validated by subjecting it to experimentation on both the lab-based prototype and the practical, real-world data centers. In addition to its technical contributions, this project involves various educational and outreach activities as well. The results of the research are integrated into the undergraduate and graduate systems courses. Finally, the toolkit, source code, datasets, and course materials developed in this project are documented and open-sourced. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.