Presentation
Range-Invariant Approximation of Non-Linear Operations for Efficient BERT Fine-Tuning
DescriptionWe propose a range-invariant look-up table (LUT) approximation that can be robustly employed for training computations of large language Transformer models, with the LUT entries significantly reduced to one entry. This method overcomes diversity across the layers and tasks with varying parameters during fine-tuning. Also, we demonstrate that the proposed methods can approximate all the non-linear operations used in BERT Encoder layers such as GELU, Layer normalization, and Softmax without degradation of scores, enabling 52% area savings in hardware implementation.
Event Type
Research Manuscript
TimeWednesday, July 12th11:10am - 11:25am PDT
Location3004, 3rd Floor
AI
AI/ML Algorithms