Tailors: Accelerating Sparse Tensor Algebra by Overbooking Buffer Capacity

Abstract

Sparse tensor algebra is a challenging class of workloads to accelerate due to few opportunities for data reuse and varying sparsity patterns. Prior sparse tensor algebra accelerators have explored tiling sparse tensors to increase exploitable data reuse and improve throughput, but typically allocate tile size in a given buffer for the worst- case number of nonzero values in a given tile. This severely limits the utilization of available memory resources and reduces data reuse. Other accelerators employ complex tiling during preprocessing or at runtime to determine the exact tile size based on its occupancy.

This thesis proposes a speculative tensor tiling approach, called overbooking, to improve buffer utilization by taking advantage of the distribution of nonzero elements in sparse tensors to construct larger tiles with greater data reuse at the cost of occasional instances where data overflows the buffer. To ensure correctness, it proposes a low-overhead hardware mechanism, Tailors, that can tolerate data overflow by design with reasonable data reuse and demonstrates that Tailors can be easily integrated into the memory hierarchy of an existing sparse tensor algebra accelerator. To ensure high buffer utilization with minimal cost to find a tile size, this thesis introduces a statistical approach, Swiftiles, to pick a tile size so that tiles usually fit within the buffer’s capacity, but can potentially overflow, i.e., it overbooks the buffers. Across a suite of 22 sparse tensor algebra workloads, the proposed overbooking strategy introduces an average speedup of 52.7× and 2.3× and an average energy reduction of 22.5× and 2.5× over ExTensor without and with optimized tiling, respectively.