Vision Mamba Distillation for Low-resolution Fine-grained Image Classification

Yao Chen, Jiabao Wang, Peichao Wang, Rui Zhang, Yang Li

Abstract

Low-resolution fine-grained image classification has recently made significant progress, largely thanks to the super-resolution techniques and knowledge distillation methods. However, these approaches lead to an exponential increase in the number of parameters and computational complexity of models. In order to solve this problem, in this letter, we propose a Vision Mamba Distillation (ViMD) approach to enhance the effectiveness and efficiency of low-resolution fine-grained image classification. Concretely, a lightweight super-resolution vision Mamba classification network (SRVM-Net) is proposed to improve its capability for extracting visual features by redesigning the classification sub-network with Mamba modeling. Moreover, we design a novel multi-level Mamba knowledge distillation loss boosting the performance, which can transfer prior knowledge obtained from a High-resolution Vision Mamba classification Network (HRVM-Net) as a teacher into the proposed SRVM-Net as a student. Extensive experiments on seven public fine-grained classification datasets related to benchmarks confirm our ViMD achieves a new state-of-the-art performance. While having higher accuracy, ViMD outperforms similar methods with fewer parameters and FLOPs, which is more suitable for embedded device applications.

Introduction

Fine-grained visual classification (FGVC) is the task of distinguishing between very similar sub-categories, such as specific bird species or car models. While modern deep learning models excel with high-resolution (HR) images, their performance significantly degrades when processing low-resolution (LR) inputs—a common challenge in real-world applications like long-range surveillance or wildlife monitoring.

Existing solutions typically rely on Super-Resolution (SR) to enhance images or Knowledge Distillation (KD) to transfer knowledge from high-res models. However, these methods are often computationally expensive, requiring high memory and processing power (FLOPs), which limits their deployment on mobile and embedded devices. To address this, we introduce Vision Mamba Distillation (ViMD), a framework that leverages the efficiency of State Space Models to achieve high accuracy with minimal computational overhead.

Methodology

The ViMD framework is built upon the Vision Mamba (Vim) architecture, which offers linear scaling and superior efficiency compared to the quadratic complexity of traditional Transformers. The architecture consists of two primary components:

  1. SRVM-Net (The Student): A lightweight network that utilizes a pre-trained Super-Resolution generator to recover image details, followed by a Vision Mamba classification sub-network.
  2. HRVM-Net (The Teacher): A high-performance Mamba network trained on high-resolution images to provide a gold-standard reference.
The architecture of the proposed ViMD framework, showing the distillation process between the teacher (HRVM-Net) and student (SRVM-Net).

A critical innovation in ViMD is the Multi-level Mamba Distillation Loss. Beyond standard Logit Distillation ($L_{LD}$), we implement Hidden States Distillation ($L_{HSD}$). This forces the student network's internal representations to align with the teacher's, ensuring the student learns the specific fine-grained features (e.g., the shape of a bird's beak) that are often lost in low-resolution textures.

Experimental Results

ViMD was evaluated across seven diverse benchmarks, including CUB-200-2011 (birds), Stanford Cars, and Oxford-IIIT Pet. To simulate real-world constraints, standard 224x224 images were downsampled to a 56x56 resolution.

Comparison of Top-1 Accuracy vs. Model Complexity. ViMD achieves the highest accuracy with the lowest parameter count and FLOPs.

Key findings include:

  • Superior Accuracy: On the challenging CUB dataset, ViMD achieved 80.19% accuracy, outperforming the previous state-of-the-art KD method (SRKD) by 2.35%.
  • Extreme Efficiency: ViMD utilizes only 6.99M parameters, a reduction of up to 95% compared to VGG16-based models (138M) and significantly lower than ResNet50-based approaches (24.6M).
  • Low Computational Cost: The model requires only 0.50G FLOPs, making it approximately 30x more efficient than VGG16-based methods and 8x more efficient than ResNet50-based methods.

Conclusion

ViMD demonstrates that Vision Mamba is a powerful and practical alternative for low-resolution fine-grained classification. By bridging the resolution gap through multi-level distillation, the framework achieves state-of-the-art results with a fraction of the hardware footprint required by traditional models. Future work will explore end-to-end training of the SR and classification components to further streamline the pipeline.

Summary updated

Abstract

Low-resolution fine-grained image classification has recently made significant progress, largely thanks to the super-resolution techniques and knowledge distillation methods. However, these approaches lead to an exponential increase in the number of parameters and computational complexity of models. In order to solve this problem, in this letter, we propose a Vision Mamba Distillation (ViMD) approach to enhance the effectiveness and efficiency of low-resolution fine-grained image classification. Concretely, a lightweight super-resolution vision Mamba classification network (SRVM-Net) is proposed to improve its capability for extracting visual features by redesigning the classification sub-network with Mamba modeling. Moreover, we design a novel multi-level Mamba knowledge distillation loss boosting the performance, which can transfer prior knowledge obtained from a High-resolution Vision Mamba classification Network (HRVM-Net) as a teacher into the proposed SRVM-Net as a student. Extensive experiments on seven public fine-grained classification datasets related to benchmarks confirm our ViMD achieves a new state-of-the-art performance. While having higher accuracy, ViMD outperforms similar methods with fewer parameters and FLOPs, which is more suitable for embedded device applications. Code is available at https://github.com/boa2004plaust/ViMD.
BibTeX
@misc{chen2024vision,
  author = {Yao Chen and Jiabao Wang and Peichao Wang and Rui Zhang and Yang Li},
  title = {{Vision Mamba Distillation for Low-resolution Fine-grained Image Classification}},
  year = {2024},
  eprint = {2411.17980},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.17980},
}
Figure