Self-knowledge distillation for fine-grained image classification

Rui Zhang, Yao Chen, Jiabao Wang, Yang Li, Xu ZhangJournal of Image and Graphics 2024 · DOI

TL;DR

Researchers have developed ECASKD, a lightweight self-knowledge distillation framework that helps small AI models learn to distinguish subtle differences in images (like bird species or car models) without needing a massive "teacher" model. By combining efficient channel attention with multi-scale feature fusion, the method achieves state-of-the-art accuracy on fine-grained tasks while remaining small enough for edge devices.

What Problem Does This Paper Address?

Fine-grained image classification (FGIC) is the "expert level" of computer vision. While a standard model might easily tell a bird from a car, an FGIC model must distinguish a California Gull from a Ring-billed Gull. These differences are often tiny—a specific beak shape or a wing pattern—hidden within large variations in pose and lighting.

To achieve high accuracy, researchers typically use massive models or "Knowledge Distillation" (KD), where a giant, pre-trained "teacher" model guides a small "student" model. However, giant teachers are expensive to train and deploy. Self-Knowledge Distillation (SKD) allows a model to be its own teacher, but existing SKD methods often struggle with FGIC because they don't focus enough on the specific, discriminative regions of an image. This paper aims to bridge that gap: creating a model that is both lightweight and "smart" enough to find the needle in the haystack.

Methods

The authors introduce ECASKD (Efficient Channel Attention based Self-Knowledge Distillation), which rests on three pillars:

  1. ECA-ResNet18 Backbone: They modified the standard ResNet18 by injecting Efficient Channel Attention (ECA) modules. Unlike traditional attention (like SE-Net) which can be computationally heavy, ECA uses a simple 1D convolution to capture dependencies between channels. This helps the model focus on important features (like the color of a bird’s crown) without bloating the parameter count.
  2. ECA-BiFPN Module: Features in a neural network exist at different scales (low-level textures vs. high-level shapes). The authors use a Weighted Bidirectional Feature Pyramid Network (BiFPN) enhanced with ECA to fuse these scales. This ensures that the "self-teacher" part of the network has a robust, multi-scale understanding of the image.
  3. Multi-level Feature Knowledge Distillation (MFKD) Loss: Instead of just comparing final predictions, the model compares its internal "attention maps" across different layers. By forcing the backbone to mimic the refined attention maps produced by the BiFPN "teacher" branch, the model learns exactly which parts of the image are most important for classification.

Datasets

The model was evaluated on three classic fine-grained benchmarks:

  • CUB-200-2011: 200 species of birds.
  • Stanford Cars: 196 types of cars.
  • FGVC-Aircraft: 100 different aircraft models.

Key Insights and Ideas

  • Attention is the Key to Distillation: The "aha moment" here is that for fine-grained tasks, the quality of the distillation depends on the model's ability to locate discriminative regions. By embedding attention directly into the distillation loss (MFKD), the model doesn't just learn what the object is, but where to look to prove it.
  • Efficiency over Complexity: The authors prove that you don't need a Transformer or a massive ResNet101 to get high accuracy. By strategically placing lightweight 1D convolutions (ECA), a ResNet18-based model can punch far above its weight class.
  • The "Self-Teacher" as a Refiner: The BiFPN branch acts as a more sophisticated version of the backbone. By distilling knowledge from this "refined" view back into the simpler backbone, the model improves itself during training and can then be deployed without the extra BiFPN overhead.

Key Results

The proposed ECASKD method significantly outperformed 15 other self-distillation methods:

  • Accuracy: Achieved 76.04% on CUB, 91.11% on Cars, and 87.64% on Aircraft. This represents a jump of 2.63% to 3.66% over previous state-of-the-art SKD methods.
  • Efficiency: The model uses only 11.9 million parameters and 2.0 GFLOPs. For comparison, it achieves accuracy on the Cars dataset that is within 0.6% of the much larger ResNet50, while using less than half the parameters and computation.
  • Visualization: Class Activation Maps (CAM) show that ECASKD focuses much more precisely on critical parts (like a bird's beak or a car's headlight) compared to baseline models, which often have "blurry" or misplaced attention.

Limitations and Future Work

While ECASKD narrows the gap, it still trails behind the absolute highest accuracy achieved by massive non-distilled models (like large-scale Swin Transformers). The authors acknowledge that while their model is optimized for "edge" deployment, there is still room to further compress the model for even more resource-constrained embedded devices. Future work may also explore combining this self-distillation approach with "label smoothing" or other regularization techniques to further boost robustness.

Summary updated

Abstract

目的在无教师模型指导的条件下,自知识蒸馏方法可以让模型从自身学习知识来提升性能,但该类方法在解决细粒度图像分类任务时,因缺乏对图像判别性区域特征的有效提取导致蒸馏效果不理想。为了解决该问题,提出了一种融合高效通道注意力的细粒度图像分类自知识蒸馏学习方法。方法首先,引入高效通道注意力(efficient channel attention, ECA)模块,设计了ECA残差模块并构建ECA-ResNet18(residual network)轻量级骨干网,用以更好地提取图像判别性区域的多尺度特征;其次,构建了高效通道注意力加权双向特征金字塔ECA-BiFPN(bidirectional feature pyramid network)模块,用以融合不同尺度的特征,构建更加鲁棒的跨尺度特征;最后,提出了一种多级特征知识蒸馏损失,用以跨尺度特征对多尺度特征的蒸馏学习。结果在Caltech-UCSD Birds 200、Stanford Cars和FGVC-Aircraft 3个公开数据集上,所提方法分别取得了76.04%、91.11%和87.64%的分类精度,与已有15种自知识蒸馏方法中最佳方法的分类精度相比,分别提高了2.63%、1.56%和3.66%。结论所提方法具有高效提取图像判别性区域特征的能力,能获得更好的细粒度图像分类精度,其轻量化的网络模型适合于面向嵌入式设备的边缘计算应用。
BibTeX
@article{zhang2024self,
  author = {Rui Zhang and Yao Chen and Jiabao Wang and Yang Li and Xu Zhang},
  title = {{Self-knowledge distillation for fine-grained image classification}},
  journal = {Journal of Image and Graphics},
  year = {2024},
  doi = {10.11834/jig.230846},
}
Figure