Partitioning models for scaling parallel sparse matrix-matrix multiplication

Date
2018
Advisor
Instructor
Source Title
ACM Transactions on Parallel Computing
Print ISSN
2329-4949
Electronic ISSN
2329-4957
Publisher
Association for Computing Machinery
Volume
4
Issue
3
Pages
13:1 - 13:34
Language
English
Type
Article
Journal Title
Journal ISSN
Volume Title
Abstract

We investigate outer-product--parallel, inner-product--parallel, and row-by-row-product--parallel formulations of sparse matrix-matrix multiplication (SpGEMM) on distributed memory architectures. For each of these three formulations, we propose a hypergraph model and a bipartite graph model for distributing SpGEMM computations based on one-dimensional (1D) partitioning of input matrices. We also propose a communication hypergraph model for each formulation for distributing communication operations. The computational graph and hypergraph models adopted in the first phase aim at minimizing the total message volume and balancing the computational loads of processors, whereas the communication hypergraph models adopted in the second phase aim at minimizing the total message count and balancing the message volume loads of processors. That is, the computational partitioning models reduce the bandwidth cost and the communication hypergraph models reduce the latency cost. Our extensive parallel experiments on up to 2048 processors for a wide range of realistic SpGEMM instances show that although the outer-product--parallel formulation scales better, the row-by-row-product--parallel formulation is more viable due to its significantly lower partitioning overhead and competitive scalability. For computational partitioning models, our experimental findings indicate that the proposed bipartite graph models are attractive alternatives to their hypergraph counterparts because of their lower partitioning overhead. Finally, we show that by reducing the latency cost besides the bandwidth cost through using the communication hypergraph models, the parallel SpGEMM time can be further improved up to 32%.

Course
Other identifiers
Book Title
Keywords
Sparse matrix-matrix multiplication, SpGEMM, Hypergraph partitioning, Graph partitioning, Communication cost, Bandwidth, Latency
Citation
Published Version (Please cite this version)