CAREER: Adaptive resource management and reconfiguration mechanisms for streaming dataflow systems

This project introduces HoloStream, a novel system that enables the efficient analysis of large data streams on different types of computing platforms. The project’s novelties include (i) a customizable design that can be tailored to various deployment settings, (ii) mechanisms to tune system configuration according to workload changes, and (iii) techniques to manage resources automatically. The project's broader significance and importance are the potential to make data streaming tools more accessible to everyday users, enabling new applications in areas like smart cities, healthcare, personalized recommendations, and tracking disease outbreaks. The project involves three sets of tasks that address challenges in adaptive resource management and efficient reconfiguration of streaming applications on dataflow systems. First, the investigator designs an adaptive distributed runtime system that can be tailored to the diverse workload characteristics of streaming applications and achieve practical performance on heterogeneous deployments. Second, she introduces online adaptation mechanisms to achieve consistent and correct reconfiguration of streaming applications without downtime. Third, the investigator develops a heterogeneity-aware optimization framework to enable self-management of streaming applications. Project results have the potential to lower the deployment costs of streaming technology for users and providers alike, and inform future research on self-management policies for long-running applications, beyond streaming analytics. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.