Incorporating Intra-Class Variance to Fine-Grained Visual Recognition

Yan Bai, Feng Gao, Yihang Lou, Shiqi Wang, Tiejun Huang, Ling-Yu DuanICME 2017 · DOI · OpenReview

TL;DR

This paper introduces Group Sensitive TRiplet Sampling (GS-TRS), a novel method that enhances fine-grained visual recognition by explicitly modeling intra-class variance. By grouping similar instances within a category and incorporating this structure into triplet loss, GS-TRS significantly improves both classification and retrieval performance on challenging datasets.

What Problem Does This Paper Address?

Fine-grained visual recognition (FGVR) aims to distinguish between visually similar categories, like different car models or bird species. While deep metric learning with triplet networks has shown promise, a key challenge remains: intra-class variance. This means that even within the same category, images can look quite different due to variations in viewpoint, lighting, or pose. Traditional triplet loss, which focuses on pulling same-class samples together and pushing different-class samples apart, doesn't adequately handle this internal variation. This can lead to suboptimal feature representations and hinder recognition accuracy. The authors aim to bridge this gap by developing a method that explicitly accounts for and leverages intra-class variance.

Methods

The core of the proposed method is the Group Sensitive TRiplet Sampling (GS-TRS) approach. Here's how it works:

  1. Intra-class Grouping: For each fine-grained category, images are first clustered into several groups. This grouping aims to capture the inherent variations within the category. The authors use a VGG-CNN_M_1024 network to extract features, followed by PCA for dimensionality reduction, and then K-means clustering to form these groups. These groups can often correspond to meaningful attributes like different viewpoints or colors.
  2. Mean-valued Triplet Loss: To mitigate the sensitivity of traditional triplet loss to anchor selection, the authors propose using a "mean-valued anchor." This anchor is computed as the average feature representation of all positive samples in a triplet. This provides a more stable and representative anchor.
  3. Intra-Class Variance Loss (ICV Loss): This is the key innovation. The ICV loss extends the triplet constraint to consider relationships between groups within the same category, in addition to the standard inter-class constraints. It enforces that samples from different groups within a category should be closer to their respective group centers than to samples from other categories, and also that samples within the same group should be even closer. This is achieved by defining two margin constraints: one for inter-class separation and another for inter-group separation.
  4. Joint Optimization: The GS-TRS triplet loss is then jointly optimized with a standard softmax classification loss. This multi-task learning approach helps to generate more discriminative feature representations, benefiting both classification and retrieval tasks. A simple linear weighting combines the two losses.

Datasets

The proposed method was evaluated on two benchmark datasets:

  • VehicleID: A large-scale dataset with over 221,000 images of 26,000 vehicles (approximately 250 models) captured by surveillance cameras. The authors treat each unique vehicle ID as a class, leading to a very fine-grained classification task. Experiments were conducted for retrieval and re-identification.
  • CompCar: A large dataset of car images primarily collected from the internet. The authors used the Part-I subset for training and evaluation, focusing on retrieval and classification.

Key Insights and Ideas

  • Intra-class variance is a critical, often overlooked, factor in FGVR: The paper convincingly demonstrates that ignoring the internal structure of a category can limit performance.
  • Grouping is a powerful way to model intra-class variance: By clustering images within a category, the method can learn to differentiate subtle variations that are crucial for fine-grained recognition.
  • Mean-valued anchors improve triplet loss stability: Using an average of positive samples as an anchor provides a more robust starting point for learning, reducing the negative impact of poorly chosen anchors.
  • Jointly optimizing classification and metric learning is beneficial: Combining softmax loss with the proposed ICV triplet loss yields superior feature representations for both tasks.

Key Results

  • Significant improvements in retrieval: On the VehicleID dataset, GS-TRS with mean-valued anchors achieved up to a 4.6% mAP improvement over previous methods on the large test set, and a substantial 23.9% mAP improvement over [19].
  • Enhanced Re-identification: For vehicle ReID on VehicleID, GS-TRS with mean-valued anchors showed a +30% improvement over Mixed Diff+CCL in the large test set.
  • Better classification accuracy: On the CompCar dataset, GS-TRS with mean-valued anchors improved classification accuracy to 79.85%, a 1.6% gain over triplet+softmax loss.
  • Qualitative improvements: Exemplar retrieval results on CompCar show that GS-TRS yields better results than traditional triplet loss, with fewer incorrect retrievals.

Limitations and Future Work

  • Grouping strategy: The authors acknowledge that optimizing the grouping strategy and the selection of anchors with respect to meaningful and effective groups is an area for future work.
  • Computational cost: While not explicitly detailed as a limitation, the grouping step and the more complex triplet sampling could potentially increase computational overhead.

The paper suggests that further research into optimizing the grouping process and anchor selection will be crucial for advancing this line of work.

Summary updated

Abstract

Fine-grained visual recognition aims to capture discriminative characteristics amongst visually similar categories. The state-of-the-art research work has significantly improved the fine-grained recognition performance by deep metric learning using triplet network. However, the impact of intra-category variance on the performance of recognition and robust feature representation has not been well studied. In this paper, we propose to leverage intra-class variance in metric learning of triplet network to improve the performance of fine-grained recognition. Through partitioning training images within each category into a few groups, we form the triplet samples across different categories as well as different groups, which is called Group Sensitive TRiplet Sampling (GS-TRS). Accordingly, the triplet loss function is strengthened by incorporating intra-class variance with GS-TRS, which may contribute to the optimization objective of triplet network. Extensive experiments over benchmark datasets CompCar and VehicleID show that the proposed GS-TRS has significantly outperformed state-of-the-art approaches in both classification and retrieval tasks.
BibTeX
@inproceedings{bai2017incorporating,
  author = {Yan Bai and Feng Gao and Yihang Lou and Shiqi Wang and Tiejun Huang and Ling-Yu Duan},
  title = {{Incorporating Intra-Class Variance to Fine-Grained Visual Recognition}},
  booktitle = {ICME},
  year = {2017},
  doi = {10.1109/icme.2017.8019371},
}
Figure