Presentation
IFHE: Intermediate-Feature Heterogeneity Enhancement for Image Synthesis in Data-Free Knowledge Distillation
DescriptionData-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. The images synthesized by existing methods cause high homogeneity on intermediate features, incurring undesired distillation performance. To address this problem, we propose the Intermediate-Feature Heterogeneity Enhancement (IFHE) method, which effectively enhances the heterogeneity of synthesized images by minimizing the loss between intermediate features and pre-set labels of synthesized images. Our IFHE outperforms the SOTA results on CIFAR-10/100 datasets of representative networks.
Event Type
Research Manuscript
TimeTuesday, July 11th3:55pm - 4:10pm PDT
Location3004, 3rd Floor
AI
AI/ML Algorithms