Data-Driven Stream Processing at the Edge

The popularity and proliferation of the Internet of Things (IoT) paradigm is resulting in a growing number of devices connected to the Internet. These devices are generating and consuming unprecedented amounts of data at the edges of the infrastructure, and are enabling new classes of data-drivenapplications, however, current approaches typically rely on cloud platforms located at the core of the infrastructure to process data. As the number of devices and the amount of data they generate and consume increases, such core-centric approaches are becoming increasingly inefficient as they need to transfer data back and forth between the edge and the core. Furthermore, the latencies associated with such data transfer may not be able to support applications involving time-critical data-driven decision making. In this paper, we propose an edge-based programming framework that allows users to define how data streams are processed based on the content and the location of the data. This enables the definition of data-driven reactive behaviors that can effectively exploit data patterns to dynamically drive stream processing, leveraging resources located at the edges of the infrastructure. We have implemented a prototype of the proposed approach and performed several experiments to evaluate its scalability and efficiency against a more typical single-cloud approach. Using a smart-city application usecase, we illustrate that the presented programming system can support data-driven stream processing using edge resources. In terms of scalability, our experiments show that the system can scale to hundreds of nodes while keeping overheads low. Our experiments also show that our approach can perform up to 56% better than a single cloud approach that does not consider data and user locality.