Monitoring Stream Properties for Continuous Query Processing

Abstract

We are developing a general-purpose Data Stream Management System for processing continuous queries over multiple continuous data streams. When a new continuous query is registered, our query optimizer creates an initial query plan (possibly merged with existing plans for previously registered queries), and allocates initial resources, such as memory for join or aggregation synopses and for input and inter-operator queues. Dynamically, plans may be altered, resources may be reallocated, and scheduling decisions are made, all based on stream data and arrival characteristics, query plan execution behavior, and resource utilization.