Presentation
CSQ: Growing Mixed-Precision Quantization Scheme with Bi-level Continuous Sparsification
DescriptionMixed-precision quantization has been widely applied on DNNs. Meanwhile, determining the precision of each layer remains challenging. Previous attempts using bit-level regularization and pruning-based precision adjustment suffered from noisy gradient and unstable convergence. We improve the bit-level mixed-precision training with a bi-level continuous relaxation on both the weight bit values and the quantization precision selection of each layer. A budget-aware regularization enables dynamic grow and prune of each layer's precision through training towards a mixed-precision quantization scheme of desired model size. The scheme enables fully differentiable training without gradient approximation, while achieving an exact quantized model in the end.
Event Type
Research Manuscript
TimeTuesday, July 11th4:40pm - 4:55pm PDT
Location3004, 3rd Floor
AI
AI/ML Algorithms
