Presentation
Libra: A Hybrid-Sparse Attention Accelerator Featuring Multi-Level Workload Balance
DescriptionTransformers have delivered exceptional performance and are widely used across various natural language processing (NLP) tasks, owing to their powerful attention mechanism. However, the high computational complexity and substantial memory usage pose significant challenges to inference efficiency. Numerous quantization and value-level sparsification methods have been proposed to overcome these challenges. Since higher sparsity leads to greater acceleration efficiency, leveraging both value-level and bit-level sparsity (hybrid sparsity) can effectively exploit the acceleration potential of the attention mechanism. However, increased sparsity exacerbates load imbalance across compute units, potentially limiting the extent of acceleration benefits. To fully exploit the acceleration potential of hybrid sparsity, we propose Libra, an attention accelerator developed through algorithm-hardware co-design. At the algorithm level, we design the bit-group-based algorithm consisting of filtered bit-group sparsification (FBS) and dynamic bit-group quantization (DBQ) to maximize the utilization of sparsity in attention. FBS imposes structured sparsity on weights, while DBQ introduces dynamic sparsification during the computation of activations. At the hardware level, we design the task pool to achieve multi-level workload balance, effectively mitigating the load imbalance among computational units induced by hybrid sparsity. Additionally, we develop an adaptive bit-width architecture to support all stages of computation in the proposed bit-group-based algorithm. Our experiments demonstrate that, compared to state-of-the-art (SOTA) attention accelerators, Libra achieves up to 1.49x ~ 5.89x speedup and 2.65\x ~ 10.82x enhancement in energy efficiency.
Event Type
Research Manuscript
TimeMonday, June 2311:45am - 12:00pm PDT
Location3001, Level 3
Similar Presentations