A novel method for scaling iterative solvers: avoiding latency overhead of parallel sparse-matrix vector multiplies
IEEE Transactions on Parallel and Distributed Systems
Institute of Electrical and Electronics Engineers
632 - 645
Item Usage Stats
In parallel linear iterative solvers, sparse matrix vector multiplication (SpMxV) incurs irregular point-to-point (P2P) communications, whereas inner product computations incur regular collective communications. These P2P communications cause an additional synchronization point with relatively high message latency costs due to small message sizes. In these solvers, each SpMxV is usually followed by an inner product computation that involves the output vector of SpMxV. Here, we exploit this property to propose a novel parallelization method that avoids the latency costs and synchronization overhead of P2P communications. Our method involves a computational and a communication rearrangement scheme. The computational rearrangement provides an alternative method for forming input vector of SpMxV and allows P2P and collective communications to be performed in a single phase. The communication rearrangement realizes this opportunity by embedding P2P communications into global collective communication operations. The proposed method grants a certain value on the maximum number of messages communicated regardless of the sparsity pattern of the matrix. The downside, however, is the increased message volume and the negligible redundant computation. We favor reducing the message latency costs at the expense of increasing message volume. Yet, we propose two iterative-improvement-based heuristics to alleviate the increase in the volume through one-to-one task-to-processor mapping. Our experiments on two supercomputers, Cray XE6 and IBM BlueGene/Q, up to 2,048 processors show that the proposed parallelization method exhibits superior scalable performance compared to the conventional parallelization method.
Inner product computation
Iterative improvement heuristic
Message latency overhead
Conjugate gradient method
Parallel processing systems
Sparse matrix-vector multiplication
Published Version (Please cite this version)http://dx.doi.org/10.1109/TPDS.2014.2311804
Showing items related by title, author, creator and subject.
Selvitopi, O.; Aykanat, Cevdet (Elsevier BV, 2016)Sparse matrix partitioning is a common technique used for improving performance of parallel linear iterative solvers. Compared to solvers used for symmetric linear systems, solvers for nonsymmetric systems offer more ...
Encapsulating multiple communication-cost metrics in partitioning sparse rectangular matrices for parallel matrix-vector multiplies Uçar, B.; Aykanat, Cevdet (SIAM, 2004)This paper addresses the problem of one-dimensional partitioning of structurally unsymmetric square and rectangular sparse matrices for parallel matrix-vector and matrix-transpose-vector multiplies. The objective is to ...
Demirbas, D.; Akturk, I.; Ozturk, O.; Güdükbay, Uğur (Oxford University Press, 2014)As a result of increasing communication demands, application-specific and scalable Network-on-Chips (NoCs) have emerged to connect processing cores and subsystems in Multiprocessor System-on-Chips. A challenge in ...