Safe data parallelism for general streaming

Date
2015
Authors
Schneider S.
Hirzel M.
Gedik, B.
Wu, Kun-Lung
Advisor
Instructor
Source Title
IEEE Transactions on Computers
Print ISSN
0018-9340
Electronic ISSN
1557-9956
Publisher
Institute of Electrical and Electronics Engineers
Volume
64
Issue
2
Pages
504 - 517
Language
English
Type
Article
Journal Title
Journal ISSN
Volume Title
Abstract

Streaming applications process possibly infinite streams of data and often have both high throughput and low latency requirements. They are comprised of operator graphs that produce and consume data tuples. General streaming applications use stateful, selective, and user-defined operators. The stream programming model naturally exposes task and pipeline parallelism, enabling it to exploit parallel systems of all kinds, including large clusters. However, data parallelism must either be manually introduced by programmers, or extracted as an optimization by compilers. Previous data parallel optimizations did not apply to selective, stateful and user-defined operators. This article presents a compiler and runtime system that automatically extracts data parallelism for general stream processing. Data-parallelization is safe if the transformed program has the same semantics as the original sequential version. The compiler forms parallel regions while considering operator selectivity, state, partitioning, and graph dependencies. The distributed runtime system ensures that tuples always exit parallel regions in the same order they would without data parallelism, using the most efficient strategy as identified by the compiler. Our experiments using 100 cores across 14 machines show linear scalability for parallel regions that are computation-bound, and near linear scalability when tuples are shuffled across parallel regions.

Course
Other identifiers
Book Title
Keywords
Data processing, Distributed computing, Data handling, Data processing, Distributed computer systems, Parallel programming, Scalability, Semantics, Data parallelism, Data parallelization, Distributed runtime, Efficient strategy, Pipeline parallelisms, Stream processing, Stream programming, Streaming applications, Program compilers
Citation
Published Version (Please cite this version)