• About
  • Policies
  • What is open access
  • Library
  • Contact
Advanced search
      View Item 
      •   BUIR Home
      • Scholarly Publications
      • Faculty of Engineering
      • Department of Computer Engineering
      • View Item
      •   BUIR Home
      • Scholarly Publications
      • Faculty of Engineering
      • Department of Computer Engineering
      • View Item
      JavaScript is disabled for your browser. Some features of this site may not work without it.

      Elastic scaling for data stream processing

      Thumbnail
      View / Download
      1.7 Mb
      Author(s)
      Gedik, B.
      Schneider S.
      Hirzel M.
      Wu, Kun-Lung
      Date
      2014
      Source Title
      IEEE Transactions on Parallel and Distributed Systems
      Print ISSN
      1045-9219
      Publisher
      IEEE Computer Society
      Volume
      25
      Issue
      6
      Pages
      1447 - 1463
      Language
      English
      Type
      Article
      Item Usage Stats
      248
      views
      634
      downloads
      Abstract
      This article addresses the profitability problem associated with auto-parallelization of general-purpose distributed data stream processing applications. Auto-parallelization involves locating regions in the application's data flow graph that can be replicated at run-time to apply data partitioning, in order to achieve scale. In order to make auto-parallelization effective in practice, the profitability question needs to be answered: How many parallel channels provide the best throughput? The answer to this question changes depending on the workload dynamics and resource availability at run-time. In this article, we propose an elastic auto-parallelization solution that can dynamically adjust the number of channels used to achieve high throughput without unnecessarily wasting resources. Most importantly, our solution can handle partitioned stateful operators via run-time state migration, which is fully transparent to the application developers. We provide an implementation and evaluation of the system on an industrial-strength data stream processing platform to validate our solution. © 1990-2012 IEEE.
      Keywords
      Data stream processing
      Elasticity
      Parallelization
      Data flow analysis
      Data flow graphs
      Profitability
      Application developers
      Auto-parallelization
      Data partitioning
      Distributed data stream processing
      Parallel channel
      Parallelizations
      Resource availability
      Data communication systems
      Permalink
      http://hdl.handle.net/11693/26675
      Published Version (Please cite this version)
      http://dx.doi.org/10.1109/TPDS.2013.295
      Collections
      • Department of Computer Engineering 1510
      Show full item record

      Browse

      All of BUIRCommunities & CollectionsTitlesAuthorsAdvisorsBy Issue DateKeywordsTypeDepartmentsCoursesThis CollectionTitlesAuthorsAdvisorsBy Issue DateKeywordsTypeDepartmentsCourses

      My Account

      Login

      Statistics

      View Usage StatisticsView Google Analytics Statistics

      Bilkent University

      If you have trouble accessing this page and need to request an alternate format, contact the site administrator. Phone: (312) 290 2976
      © Bilkent University - Library IT

      Contact Us | Send Feedback | Off-Campus Access | Admin | Privacy