Autopipelining for data stream processing
IEEE Transactions on Parallel and Distributed Systems
Please cite this item using this persistent URLhttp://hdl.handle.net/11693/20712
Stream processing applications use online analytics to ingest high-rate data sources, process them on-the-fly, and generate live results in a timely manner. The data flow graph representation of these applications facilitates the specification of stream computing tasks with ease, and also lends itself to possible runtime exploitation of parallelization on multicore processors. While the data flow graphs naturally contain a rich set of parallelization opportunities, exploiting them is challenging due to the combinatorial number of possible configurations. Furthermore, the best configuration is dynamic in nature; it can differ across multiple runs of the application, and even during different phases of the same run. In this paper, we propose an autopipelining solution that can take advantage of multicore processors to improve throughput of streaming applications, in an effective and transparent way. The solution is effective in the sense that it provides good utilization of resources by dynamically finding and exploiting sources of pipeline parallelism in streaming applications. It is transparent in the sense that it does not require any hints from the application developers. As a part of our solution, we describe a light-weight runtime profiling scheme to learn resource usage of operators comprising the application, an optimization algorithm to locate best places in the data flow graph to explore additional parallelism, and an adaptive control scheme to find the right level of parallelism. We have implemented our solution in an industrial-strength stream processing system. Our experimental evaluation based on microbenchmarks, synthetic workloads, as well as real-world applications confirms that our design is effective in optimizing the throughput of stream processing applications without requiring any changes to the application code. © 1990-2012 IEEE.
- Research Paper 
Showing items related by title, author, creator and subject.
Serel, D.A. (2009)For monitoring the stability of a process, various control charts based on exponentially weighted moving average (EWMA) statistics have been proposed in the literature. We study the economic design of EWMA-based mean and ...
Akturk, M.S.; Avci, S. (1996)In view of the high investment and tooling cost of a CNC machining centre, the cutting and idle times should be optimised by considering the tool consumption and the non-machining time cost components. In this paper, we ...
Şaykol, E.; Sinop, A.K.; Güdükbay, U.; Ulusoy Ö.; Çetin, A.E. (2004)There is an accelerating demand to access the visual content of documents stored in historical and cultural archives. Availability of electronic imaging tools and effective image processing techniques makes it feasible to ...