Joker: elastic stream processing with organic adaptation

Simple item page
buir.contributor.authorKahveci, Basri
buir.contributor.authorGedik, Buğra
dc.citation.epage223en_US
dc.citation.spage205en_US
dc.citation.volumeNumber137en_US
dc.contributor.authorKahveci, Basrien_US
dc.contributor.authorGedik, Buğraen_US
dc.date.accessioned2021-02-25T06:56:49Z
dc.date.available2021-02-25T06:56:49Z
dc.date.issued2020
dc.departmentDepartment of Computer Engineeringen_US
dc.description.abstractThis paper addresses the problem of auto-parallelization of streaming applications. We propose an online parallelization optimization algorithm that adjusts the degree of pipeline and data parallelism in a joint manner. We define an operator development API and a flexible parallel execution model to form a basis for the optimization algorithm. The operator interface unifies the development of different types of operators and makes operator properties visible in order to enable safe optimizations. The parallel execution model splits a data flow graph into regions. A region contains the longest sequence of compatible operators that are amenable to data parallelism as a whole and can be further parallelized with pipeline parallelism. We also develop a stream processing run-time, named Joker, to scale the execution of streaming applications in a safe, transparent, dynamic, and automatic manner. This ability is called organic adaptation. Joker implements the runtime machinery to execute a data flow graph with any parallelization configuration and most importantly change this configuration at run-time with low cost in the presence of partitioned stateful operators, in a way that is transparent to the application developers. Joker continuously monitors the run-time performance, and runs the optimization algorithm to resolve bottlenecks and scale the application by adjusting the degree of pipeline and data parallelism. The experimental evaluation based on micro-benchmarks and real-world applications showcase that our solution accomplishes elasticity by finding an effective parallelization configuration.en_US
dc.description.provenanceSubmitted by Onur Emek (onur.emek@bilkent.edu.tr) on 2021-02-25T06:56:49Z No. of bitstreams: 1 Joker_Elastic_stream_processing_with_organic_adaptation.pdf: 1389546 bytes, checksum: 0ac753326535cf683b27c7c6390131cc (MD5)en
dc.description.provenanceMade available in DSpace on 2021-02-25T06:56:49Z (GMT). No. of bitstreams: 1 Joker_Elastic_stream_processing_with_organic_adaptation.pdf: 1389546 bytes, checksum: 0ac753326535cf683b27c7c6390131cc (MD5) Previous issue date: 2020en
dc.description.sponsorshipThis work was supported by the Scientific and Technical Re-search Council of Turkey (TUBITAK) under Grant No. 114E472.en_US
dc.embargo.release2022-03-01
dc.identifier.doi10.1016/j.jpdc.2019.10.012en_US
dc.identifier.issn0743-7315
dc.identifier.urihttp://hdl.handle.net/11693/75576
dc.language.isoEnglishen_US
dc.publisherElsevieren_US
dc.relation.isversionofhttps://dx.doi.org/10.1016/j.jpdc.2019.10.012en_US
dc.source.titleJournal of Parallel and Distributed Computingen_US
dc.subjectStream processingen_US
dc.subjectElasticityen_US
dc.subjectParallelizationen_US
dc.titleJoker: elastic stream processing with organic adaptationen_US
dc.typeArticleen_US

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Joker_Elastic_stream_processing_with_organic_adaptation.pdf
Size:
1.33 MB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Scholarly Publications - Computer Engineering