Pipelined fission for stream programs with dynamic selectivity and partitioned state
Author
Özsema, Habibe Güldamla
Advisor
Güdükbay, Uğur
Date
2014-12Publisher
Bilkent University
Language
English
Type
ThesisItem Usage Stats
82
views
views
26
downloads
downloads
Abstract
There is an ever increasing rate of digital information available in the form of
online data streams. In many application domains, high throughput processing
of such data is a critical requirement for keeping up with the soaring input rates.
Data stream processing is a computational paradigm that aims at addressing this
challenge by processing data streams in an on-the-fly manner.
In this thesis, we study the problem of automatically parallelizing data stream
processing applications to improve throughput. The parallelization is automatic
in the sense that stream programs are written sequentially by the application
developers and are parallelized by the system. We adopt the asynchronous data
flow model for our work, where operators often have dynamic selectivity and are
stateful. We solve the problem of pipelined fission, in which the original sequential
program is parallelized by taking advantage of both pipeline and data parallelism
at the same time. Our solution supports partitioned stateful data parallelism
with dynamic selectivity and is designed for shared-memory multi-core machines.
We first develop a cost-based formulation to express pipelined fission as an optimization
problem. The bruteforce solution of this problem takes a very long
time for moderately sized stream programs. Accordingly, we develop a heuristic
algorithm that can quickly, but approximately, solve this problem. We provide
an extensive evaluation studying the performance of our solution, including simulations
and experiments with an industrial-strength Data Stream Processing
Systems (DSPS). Our results show good scalability for applications that contain
sufficient parallelism, closeness to optimal performance for the algorithm.
Permalink
http://hdl.handle.net/11693/14014Collections
The following license files are associated with this item: