Partitioning functions for steteful data parallelism in stream processing

Date
2014
Authors
Gedik, B.
Editor(s)
Advisor
Supervisor
Co-Advisor
Co-Supervisor
Instructor
Source Title
The VLDB Journal
Print ISSN
1066-8888
Electronic ISSN
Publisher
Association for Computing Machinery
Volume
23
Issue
4
Pages
517 - 539
Language
English
Journal Title
Journal ISSN
Volume Title
Series
Abstract

In this paper we study partitioning functions for stream processing systems that employ stateful data parallelism to improve application throughput. In particular, we develop partitioning functions that are effective under workloads where the domain of the partitioning key is large and its value distribution is skewed. We define various desirable properties for partitioning functions, ranging from balance properties such as memory, processing, and communication balance, structural properties such as compactness and fast lookup, and adaptation properties such as fast computation and minimal migration. We introduce a partitioning function structure that is compact and develop several associated heuristic construction techniques that exhibit good balance and low migration cost under skewed workloads. We provide experimental results that compare our partitioning functions to more traditional approaches such as uniform and consistent hashing, under different workload and application characteristics, and show superior performance.

Course
Other identifiers
Book Title
Citation
Published Version (Please cite this version)