Building user-defined runtime adaptation routines for stream processing applications

Date
2012
Authors
Jacques-Silva, G.
Gedik, B.
Wagle, R.
Wu, Kun-Lung
Kumar, V.
Advisor
Instructor
Source Title
Proceedings of the VLDB Endowment
Print ISSN
2150-8097
Electronic ISSN
Publisher
VLDB Endowment
Volume
5
Issue
12
Pages
1826 - 1837
Language
English
Type
Article
Journal Title
Journal ISSN
Volume Title
Abstract

Stream processing applications are deployed as continuous queries that run from the time of their submission until their cancellation. This deployment mode limits developers who need their applications to perform runtime adaptation, such as algorithmic adjustments, incremental job deployment, and application-specific failure recovery. Currently, developers do runtime adaptation by using external scripts and/or by inserting operators into the stream processing graph that are unrelated to the data processing logic. In this paper, we describe a component called orchestrator that allows users to write routines for automatically adapting the application to runtime conditions. Developers build an orchestrator by registering and handling events as well as specifying actuations. Events can be generated due to changes in the system state (e.g., application component failures), built-in system metrics (e.g., throughput of a connection), or custom application metrics (e.g., quality score). Once the orchestrator receives an event, users can take adaptation actions by using the orchestrator actuation APIs. We demonstrate the use of the orchestrator in IBM's System S in the context of three different applications, illustrating application adaptation to changes on the incoming data distribution, to application failures, and on-demand dynamic composition. © 2012 VLDB Endowment.

Course
Other identifiers
Book Title
Keywords
Application adaptation, Application components, Application failure, Built - in system, Continuous queries, Data distribution, Dynamic composition, Failure recovery, Runtime adaptation, Runtimes, Stream processing, System state, Data processing
Citation
Published Version (Please cite this version)