Group Softmax Loss With Discriminative Feature Grouping

Takumi KobayashiWACV 2021

TL;DR

This paper introduces a novel "Group Softmax Loss" that encourages deep neural networks to learn more discriminative feature representations, especially when training data is scarce. By intelligently grouping feature components and applying a modified softmax loss, the method effectively mitigates overfitting and improves classification performance across various datasets.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have achieved remarkable success in various tasks, but their effectiveness hinges on vast amounts of labeled data. The standard softmax cross-entropy loss, while widely used, often struggles to train DNNs effectively on smaller datasets due to overfitting. This limitation arises because the standard softmax loss treats all feature components equally, potentially allowing a few highly discriminative features to dominate the learning process while neglecting others. This paper aims to address this by developing a loss function that promotes discriminative learning across all feature components, even when data is limited.

Methods

The core innovation is the Group Softmax Loss coupled with Discriminative Feature Grouping.

  • Group Softmax Loss: Instead of applying the standard softmax loss to the entire feature vector, this method partitions the feature components into several groups. A modified softmax loss is then computed for each group. Two types of group softmax losses are proposed:
  • Type-I Loss: Introduces a scaling factor to balance the magnitudes of group logits, similar to dropout.
  • Type-II Loss: A simpler composition of group-wise softmax losses, which theoretically provides a tighter upper bound on the standard softmax loss. These group losses are designed to be upper bounds of the standard softmax loss, meaning minimizing them can lead to improved performance.

  • Discriminative Feature Grouping: To effectively group features, the paper proposes a method to measure the "discriminative power" of each individual feature component. This is achieved by calculating a discriminativity score based on the difference between the ingredient logits for the ground-truth class and other classes. Features are then sorted based on this score, and groups are formed by partitioning these sorted features. This ensures that less discriminative features are grouped together, allowing the group loss to focus on improving their learning without being overshadowed by highly discriminative features. This grouping is performed efficiently within mini-batches.

The paper also frames this approach as a form of adversarial regularization at the loss level, where the grouping mechanism adversarially influences the feature learning process.

Datasets

The proposed method was evaluated on the following datasets:

  • Food-101: A middle-scale dataset with 101 food categories, used for ablation studies.
  • ImageNet: A large-scale dataset used for analyzing performance across varying numbers of classes and different CNN architectures.
  • Caltech-256: A smaller-scale dataset used for evaluating performance on limited data.
  • SUN-397: Another smaller-scale dataset for evaluating performance on limited data.

No new benchmarks were introduced; existing datasets were used to demonstrate the effectiveness of the proposed method.

Key Insights and Ideas

  • Feature-level discriminativity is key: The paper highlights that focusing on the discriminativity of individual feature components, rather than just the overall feature vector, is crucial for effective learning, especially with limited data.
  • Grouping as a regularization technique: The group softmax loss, particularly when combined with discriminative grouping, acts as a form of adversarial regularization, pushing the network to learn more robust features and preventing overfitting.
  • Exposing non-discriminative features: By separating less discriminative features into their own groups, the loss function can specifically encourage their improvement, leading to a more balanced and comprehensive feature representation.
  • Theoretical connection to adversarial training: The proposed method can be viewed as an adversarial training approach operating at the loss function level, offering a stable and effective way to improve generalization.

Key Results

  • Improved performance on Food-101: The proposed Group Softmax Loss (especially Type-II) significantly reduced the error rate compared to the standard softmax loss, achieving an error rate of 16.10%±0.29 with 4 groups, compared to 19.67%±0.48 for standard softmax.
  • Superiority of discriminative grouping: Discriminative feature grouping consistently outperformed random or pre-fixed grouping strategies across experiments.
  • Effective on ImageNet: The method showed performance improvements on ImageNet, particularly with varying numbers of classes, with Type-II loss outperforming Type-I on smaller class subsets.
  • Outperforms hand-crafted methods on smaller datasets: On Caltech-256 and SUN-397, the proposed method, when applied to CNNs, outperformed traditional hand-crafted feature methods.
  • Competitive with regularization methods: While not always outperforming methods like Dropout, the proposed loss offers a complementary approach that leverages all feature components rather than dropping them.

Limitations and Future Work

  • Number of groups (G): While the paper integrates multiple group losses to handle varying numbers of groups, determining the optimal number of groups for a specific task might require further exploration.
  • Computational cost: Although the paper claims the computational cost is negligible, the grouping process and multiple softmax computations add a slight overhead compared to the standard softmax loss.
  • Future work: The authors suggest combining their method with data augmentation techniques and exploring its integration with other loss variants (e.g., large-margin losses) and regularization methods. They also note that the proposed method tackles the data-hunger issue, making it a promising direction for future research.

Summary updated

BibTeX
@inproceedings{kobayashi2021group,
  author = {Takumi Kobayashi},
  title = {{Group Softmax Loss With Discriminative Feature Grouping}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}

Loading related content...

Figure