A parallel fill estimation algorithm for sparse matrices and tensors in blocked formats

Abstract

Many sparse matrices and tensors from a variety of applications, such as finite element methods and computational chemistry, have a natural aligned rectangular nonzero block structure. Researchers have designed high-performance blocked sparse operations which can take advantage of this sparsity structure to reduce the complexity of storing the locations of nonzeros. The performance of a blocked sparse operation depends on how well the block size reflects the structure of nonzeros in the tensor. Sparse tensor structure is generally unknown until runtime, so block size selection must be efficient. The fill is a quantity which, for some block size, relates the number of nonzero blocks to the number of nonzeros. Many performance models use the fill to help choose a block size. However, the fill is expensive to compute exactly. We present a sampling-based algorithm called Phil to estimate the fill of sparse matrices and tensors in any format. We provide theoretical guarantees for sparse matrices and tensors, and experimental results for matrices. The existing state-of-the-art fill estimation algorithm, which we will call OSKI, runs in time linear in the number of elements in the tensor. The number of samples Phil needs to compute a fill estimate is unrelated to the number of nonzeros and depends only on the order (number of dimensions) of the tensor, desired accuracy of the estimate, desired probability of achieving this accuracy, and number of considered block sizes. We parallelize Phil, and refer to the parallel implementation as PPhil. We compare Phil, PPhil, and OSKI on a suite of 42 matrices. On average, PPhil was able to produce a fill estimate in 1.3810 times the time it took to compute one sparse matrix vector multiply, which was 61.176 times faster than OSKI. The maximum error generated by Phil was 0.0480, while OSKI sometimes produced estimates with a complete loss of accuracy. Finally, we find that Phil and OSKI produce comparable speedups in multicore blocked sparse matrix-vector multiplication (SpMV) when the block size was chosen using fill estimates in a model due to Vuduc et al. Much of the work presented in this thesis appears in ["A Fill Estimation Algorithm for Sparse Matrices and Tensors in Blocked Formats," in 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS), May 2018, pp. 546-556.], a paper coauthored with Helen Xu and Nicholas Schiefer. The parallel algorithm PPhil and its implementation are novel contributions of this thesis. Helen's masters thesis is also based on the IPDPS publication, and adds additional test matrices ["Fill Estimation for Blocked Sparse Matrices and Tensors," Master's thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Jun. 2018.].