Abstract
In this work, we present a novel mask guided attention (MGA) method for fine-grained patchy image classification. The key challenge of fine-grained patchy image classification lies in two folds: ultra-fine-grained inter-category variances among objects and very few data available for training. This motivates us to consider employing more useful supervision signals to train a discriminative model within limited training samples. Specifically, the proposed MGA integrates a pre-trained semantic segmentation model that produces an auxiliary supervision signal, i.e., a patchy attention mask, enabling discriminative representation learning. The patchy attention mask drives the classifier to filter out the insignificant parts of images, which enhances the robustness of MGA. Experimental results demonstrate that our MGA method achieves superior performance on three datasets compared with state-of-the-art methods, improving accuracy by up to 2.25%.
Introduction
Fine-grained image classification involves distinguishing between categories that are visually very similar. "Patchy" image classification represents an extreme version of this task, where the differences are microscopic—such as the specific vein patterns on a leaf—and the available training data is extremely scarce. Standard deep learning models often struggle here because they may focus on irrelevant background noise or overfit on the small number of available samples. The proposed Mask Guided Attention (MGA) framework addresses these issues by using a two-stage process that explicitly directs the model's attention to the most informative regions of an image.
Methodology
The MGA framework leverages auxiliary supervision to guide the model's focus, preventing it from getting trapped in local optima or memorizing irrelevant pixels.
The framework consists of three main components:
- Mask Generation: A pre-trained semantic segmentation model (HMSANet) is used to generate a "patchy attention mask." For example, in leaf classification, this mask highlights the primary and secondary veins.
- Guided Attention: Instead of letting the model learn attention implicitly, the generated mask is used as ground truth to supervise the attention map's learning via a Mean Squared Error (MSE) loss. This "forces" the model to focus on discriminative regions like veins.
- Feature Fusion: The model extracts features from the original image and combines them with "attended" features. This ensures the final representation captures both global context and high-priority details.
Datasets and Experimental Results
The MGA method was evaluated on three specialized "patchy" datasets: SoyCultivarVein (200 soybean cultivars), HainanLeaf (20 categories), and BtfPIS (butterfly wing patches).
Key results include:
- Performance Gains: MGA consistently outperformed standard state-of-the-art CNNs. On the SoyCultivarVein dataset, it achieved 45.75% accuracy, a significant improvement over baseline methods.
- Robustness: The method proved effective even when the segmentation masks were not perfect. On the HainanLeaf dataset, despite low segmentation quality (41.31% mIoU), the classifier reached 90% accuracy.
- Ablation Study: Removing the mask supervision resulted in a 2-2.25% drop in accuracy, confirming that explicit guidance is a critical component for success on small, fine-grained datasets.
Conclusion
Mask Guided Attention provides a robust solution for ultra-fine-grained classification tasks where data is limited. By providing an auxiliary supervision signal, the model avoids the common pitfall of memorizing irrelevant pixels and instead learns to focus on the structural features that define the categories. Future work will explore extending this guided attention mechanism to other domains with similar "patchy" characteristics.
Summary updated