Elastic scaling for data stream processing

Date
2014
Authors
Gedik, B.
Schneider S.
Hirzel M.
Wu, Kun-Lung
Editor(s)
Advisor
Supervisor
Co-Advisor
Co-Supervisor
Instructor
Source Title
IEEE Transactions on Parallel and Distributed Systems
Print ISSN
1045-9219
Electronic ISSN
Publisher
IEEE Computer Society
Volume
25
Issue
6
Pages
1447 - 1463
Language
English
Journal Title
Journal ISSN
Volume Title
Series
Abstract

This article addresses the profitability problem associated with auto-parallelization of general-purpose distributed data stream processing applications. Auto-parallelization involves locating regions in the application's data flow graph that can be replicated at run-time to apply data partitioning, in order to achieve scale. In order to make auto-parallelization effective in practice, the profitability question needs to be answered: How many parallel channels provide the best throughput? The answer to this question changes depending on the workload dynamics and resource availability at run-time. In this article, we propose an elastic auto-parallelization solution that can dynamically adjust the number of channels used to achieve high throughput without unnecessarily wasting resources. Most importantly, our solution can handle partitioned stateful operators via run-time state migration, which is fully transparent to the application developers. We provide an implementation and evaluation of the system on an industrial-strength data stream processing platform to validate our solution. © 1990-2012 IEEE.

Course
Other identifiers
Book Title
Citation
Published Version (Please cite this version)