CAREER: Scalable and Adaptive Edge Stream Processing

Internet-of-Things (IoT) applications such as self-driving cars, augmented reality, interactive gaming, and event monitoring have a tremendous potential to improve our lives. These applications generate a large influx of sensor data at massive scales. Under many time-critical scenarios, these massive data streams must be processed in a very short time to derive actionable intelligence. This CAREER project aims to support time-critical IoT applications by applying the stream processing paradigm to the Edge computing architecture. The success of this research will benefit many time-critical IoT applications in the areas such as factory automation, the tactile internet, autonomous vehicles, and process automation. It will also substantially improve the performance profiles of a variety of data processing systems, such as wide-area data analytics systems, mobile data access systems, event tracking systems, and streaming databases. As an integral part of its research program, this CAREER project involves K-12, undergraduate and graduate level education in partnership with the local Public School system. Specifically, this CAREER project will build a scalable and adaptive Edge stream processing engine, which enables fast stream processing of a large number of concurrent IoT queries in the dynamic, heterogeneous Edge environment. This work includes three primary research directions. First, a new dynamic dataflow graph abstraction will be implemented, which automatically chains, parallelizes and replicates stream operators to adapt to the Edge dynamics and handle failures in a scalable way. Second, a new customizable data shuffling service abstraction will be implemented, which customizes the data shuffling path (e.g., ring shuffle, hierarchical tree shuffle, butterfly wrap shuffle) at runtime for the given network topology and workload. Third, a fully decentralized architecture with many distributed schedulers will be implemented, in which each scheduler operates autonomously to process IoT queries. All three parts of the project will be prototyped and implemented on real-world stream processing systems and validated by performing real-world experiments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.