Three-branch and Mutil-scale learning for Fine-grained Image Recognition (TBMSL-Net)

Fan Zhang, Guisheng Zhai, Meng Li, Yizhao Liu

Abstract

Fine-grained visual categorization (FGVC) is a challenging task due to small inter-class variations and large intra-class variations. We propose a multi-branch and multi-scale learning network (MMAL-Net, also referred to as TBMSL-Net) that automatically identifies discriminative object and part regions without requiring manual bounding-box or part annotations. By leveraging an Attention Object Location Module (AOLM) and an Attention Part Proposal Module (APPM), the network processes images at multiple scales through a shared-parameter CNN backbone. Our approach achieves state-of-the-art performance on the CUB-200-2011, FGVC-Aircraft, and Stanford Cars datasets while maintaining high inference efficiency.

Introduction

General object recognition often focuses on broad categories (e.g., "bird" vs. "car"). In contrast, Fine-Grained Visual Categorization (FGVC) requires distinguishing between highly similar sub-categories, such as specific aircraft models or bird species. This task is complicated by low inter-class variation—where different species look nearly identical—and high intra-class variation caused by diverse poses, lighting, and backgrounds.

Traditional methods often relied on labor-intensive manual annotations (bounding boxes and part locations) to guide the model toward "discriminative regions" like the shape of a beak or the pattern on a wing. TBMSL-Net addresses this by using attention mechanisms to automatically locate these regions, enabling end-to-end training using only image-level labels.

Methodology

The proposed architecture utilizes a three-branch design built upon a shared ResNet-50 backbone. This allows the model to learn from global, object-level, and part-level features simultaneously without a significant increase in parameter count.

The MMAL-Net architecture featuring Raw, Object, and Part branches.

Attention-Driven Modules

  • Raw Branch: Processes the full image to understand global characteristics.
  • Object Branch (AOLM): The Attention Object Location Module aggregates feature maps from the CNN to create an activation map. By identifying where neurons fire most strongly across multiple layers, it generates a mask to crop the object from the background, providing a "finer scale" view of the target's structure.
  • Part Branch (APPM): The Attention Part Proposal Module uses a sliding window approach on the feature maps to find regions with high information density. It employs Non-Maximum Suppression (NMS) to ensure the selected parts (e.g., a bird's head or wing) are distinct and informative.

Crucially, all three branches share the same parameters. This weight sharing ensures that learning from one scale (like a close-up of a part) benefits the others, improving overall robustness.

Experimental Results

The model was evaluated on three gold-standard FGVC datasets: CUB-200-2011 (200 bird species), FGVC-Aircraft (100 aircraft variants), and Stanford Cars (196 car models).

Comparison of the proposed method (Ours) against state-of-the-art methods across three datasets.

As shown in the results table, the proposed method consistently outperforms existing architectures. It achieved an accuracy of 89.6% on CUB-200-2011, 94.7% on FGVC-Aircraft, and 95.0% on Stanford Cars.

Beyond accuracy, the model demonstrates significant efficiency. During the testing phase, the "Part" branch can be discarded, as the "Object" branch alone provides optimal classification performance. This results in an inference time of just 1.80 ms per image on a Tesla P100, roughly three times faster than competing models like NTS-Net.

Conclusion

TBMSL-Net provides a robust and efficient solution for fine-grained recognition by automatically focusing on multi-scale discriminative regions. By eliminating the need for manual part annotations and optimizing for fast inference, it bridges the gap between high-accuracy research models and practical applications. Future work will explore adaptive window selection for the part proposal module to further enhance flexibility across diverse image types.

Summary updated

Abstract

ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is one of the most authoritative academic competitions in the field of Computer Vision (CV) in recent years. But applying ILSVRC's annual champion directly to fine-grained visual categorization (FGVC) tasks does not achieve good performance. To FGVC tasks, the small inter-class variations and the large intra-class variations make it a challenging problem. Our attention object location module (AOLM) can predict the position of the object and attention part proposal module (APPM) can propose informative part regions without the need of bounding-box or part annotations. The obtained object images not only contain almost the entire structure of the object, but also contains more details, part images have many different scales and more fine-grained features, and the raw images contain the complete object. The three kinds of training images are supervised by our multi-branch network. Therefore, our multi-branch and multi-scale learning network(MMAL-Net) has good classification ability and robustness for images of different scales. Our approach can be trained end-to-end, while provides short inference time. Through the comprehensive experiments demonstrate that our approach can achieves state-of-the-art results on CUB-200-2011, FGVC-Aircraft and Stanford Cars datasets. Our code will be available at this https URL
BibTeX
@misc{zhang2020three,
  author = {Fan Zhang and Guisheng Zhai and Meng Li and Yizhao Liu},
  title = {{Three-branch and Mutil-scale learning for Fine-grained Image Recognition (TBMSL-Net)}},
  year = {2020},
  eprint = {2003.09150},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.09150},
}
Figure