Fast and efficient model parallelism for deep convolutional neural networks

Simple item page
buir.advisorÖzdal, Muhammet Mustafa
dc.contributor.authorEserol, Burak
dc.date.accessioned2019-08-23T06:41:05Z
dc.date.available2019-08-23T06:41:05Z
dc.date.copyright2019-08
dc.date.issued2019-08
dc.date.submitted2019-08-21
dc.departmentDepartment of Computer Engineeringen_US
dc.descriptionCataloged from PDF version of article.en_US
dc.descriptionThesis (M.S.): Bilkent University, Department of Computer Engineering, İhsan Doğramacı Bilkent University, 2019.en_US
dc.descriptionIncludes bibliographical references (leaves 72-76).en_US
dc.description.abstractConvolutional Neural Networks (CNNs) have become very popular and successful in recent years. Increasing the depth and number of parameters of CNNs has crucial importance on this success. However, it is hard to t deep convolutional neural networks into a single machine's memory and it takes a very long time to train these deep convolutional neural networks. There are two parallelism methods to solve this problem: data parallelism and model parallelism. In data parallelism, the neural network model is replicated among different machines and data is partitioned among them. Each replica trains its data and communicates parameters and their gradients with other replicas. This process results in a huge communication volume in data parallelism, which slows down the training and convergence of the deep neural network. In model parallelism, a deep neural network model is partitioned among different machines and trained in a pipelined manner. However, it requires a human expert to partition the network and it is hard to obtain low communication volume as well as a low computational load balance ratio by using known partitioning methods. In this thesis, a new model parallelism method called hypergraph partitioned model parallelism is proposed. It does not require a human expert to partition the network and obtains a better computational load balance ratio along with better communication volume compared to the existing model parallelism techniques. Besides, the proposed method also reduces the communication volume overhead in data parallelism by 93%. Finally, it is also shown that distributing a deep neural network using the proposed hypergraph partitioned model rather than the existing parallelism methods causes the network to converge faster to the target accuracy.en_US
dc.description.degreeM.S.en_US
dc.description.statementofresponsibilityby Burak Eserolen_US
dc.embargo.release2020-02-19
dc.format.extentxvi, 81 leaves : charts (some color) ; 30 cm.en_US
dc.identifier.itemidB106908
dc.identifier.urihttp://hdl.handle.net/11693/52360
dc.language.isoEnglishen_US
dc.publisherBilkent Universityen_US
dc.rightsinfo:eu-repo/semantics/openAccessen_US
dc.subjectParallel and distributed deep learningen_US
dc.subjectConvolutional Neural Networksen_US
dc.subjectModel parallelismen_US
dc.subjectData parallelismen_US
dc.titleFast and efficient model parallelism for deep convolutional neural networksen_US
dc.title.alternativeDerin konvolüsyonel sinir ağları için hızlı ve verimli model paralelleştirmesien_US
dc.typeThesisen_US

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
BurakEserol_Thesis.pdf
Size:
1.08 MB
Format:
Adobe Portable Document Format
Description:
Full printable version
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Dept. of Computer Engineering - Master's degree