Discriminability-enforcing loss to improve representation learning

Florinel-Alin Croitoru, Diana-Nicoleta Grigore, Radu Tudor Ionescu

Abstract

Standard training of deep neural networks using cross-entropy loss often results in high-level features that lack specialization or are unevenly distributed across target classes. This paper introduces a novel training objective that incorporates two auxiliary loss terms: a Gini impurity loss to enforce feature specialization and a Kullback–Leibler (KL) divergence loss to ensure a balanced distribution of these specialized features. Empirical evaluations on CIFAR-100 and cross-domain tasks using Caltech 101 demonstrate that this approach consistently enhances model accuracy and stability across various architectures, including ResNets and Transformers, with zero overhead during inference.

Introduction

In deep learning, the internal representations of a model are typically treated as emergent properties of the backpropagation process. While effective, standard categorical cross-entropy loss does not explicitly regulate the behavior of individual high-level features. This can lead to "feature ambiguity," where features respond weakly to multiple classes, or "feature imbalance," where the network over-allocates resources to easily distinguishable categories. To address these issues, we propose a method to explicitly enforce discriminability during the training phase.

Methodology

The proposed framework augments the standard cross-entropy loss ($L_*$) with two novel auxiliary terms designed to shape the feature space.

Gini Impurity Loss ($L_{Gini}$)

Inspired by decision tree algorithms, the Gini loss measures the "purity" of feature activation maps. By minimizing the Gini impurity for each high-level feature, the model is incentivized to make each feature highly specialized—ideally activating strongly for a single class and remaining dormant for others.

KL Divergence Loss ($L_{KL}$)

To prevent the model from over-specializing in a subset of classes, the KL divergence loss acts as a regularizer. It minimizes the divergence between the distribution of feature specializations and the actual class distribution in the dataset. This ensures that the "feature budget" is distributed equitably across all target categories, preventing the neglect of harder-to-learn classes.

The final objective function is defined as a weighted sum: $L_{total} = L_* + \lambda_1 L_{Gini} + \lambda_2 L_{KL}$. Since these terms are only applied during training, the resulting models maintain their original inference speed.

Experimental Results

The efficacy of the discriminability-enforcing loss was validated on the CIFAR-100 benchmark and a cross-domain subset of Caltech 101.

Performance Gains

On CIFAR-100, the method provided consistent improvements across multiple architectures. For instance, a ResNet-18 model saw an accuracy increase from 71.47% to 72.34%. Notably, a lighter ResNet-17 model trained with the proposed loss (73.00%) outperformed a standard ResNet-18, suggesting that optimized feature learning can be more effective than simply increasing model depth.

Qualitative Analysis

The impact of the proposed loss is most evident in challenging classification scenarios. As shown in the comparison below, the model (Ours) correctly identifies objects that the baseline model confuses, such as distinguishing between a lobster and a crab or identifying a butterfly-shaped lollipop.

Qualitative comparison of classification results between the baseline and the proposed method (Ours) against ground-truth labels.

Cross-Domain Generalization

To test the robustness of the learned representations, models were evaluated on a transfer task from CIFAR-100 to Caltech 101. The proposed method improved ResNet-18 performance from 63.04% to 64.33%, indicating that enforcing discriminability helps the network capture more generalizable semantic features rather than dataset-specific noise.

Conclusion

By explicitly enforcing feature specialization and balance through Gini and KL divergence losses, we can significantly improve the representation learning capabilities of neural networks. The synergy between these two terms is crucial: specialization provides clarity, while balance ensures coverage. This approach offers a computationally efficient way to boost performance across diverse datasets and architectures.

Summary updated

Abstract

During the training process, deep neural networks implicitly learn to represent the input data samples through a hierarchy of features, where the size of the hierarchy is determined by the number of layers. In this paper, we focus on enforcing the discriminative power of the high-level representations, that are typically learned by the deeper layers (closer to the output). To this end, we introduce a new loss term inspired by the Gini impurity, which is aimed at minimizing the entropy (increasing the discriminative power) of individual high-level features with respect to the class labels. Although our Gini loss induces highly-discriminative features, it does not ensure that the distribution of the high-level features matches the distribution of the classes. As such, we introduce another loss term to minimize the Kullback-Leibler divergence between the two distributions. We conduct experiments on two image classification data sets (CIFAR-100 and Caltech 101), considering multiple neural architectures ranging from convolutional networks (ResNet-17, ResNet-18, ResNet-50) to transformers (CvT). Our empirical results show that integrating our novel loss terms into the training objective consistently outperforms the models trained with cross-entropy alone, without increasing the inference time at all.
BibTeX
@inproceedings{croitoru2022discriminability,
  author = {Florinel-Alin Croitoru and Diana-Nicoleta Grigore and Radu Tudor Ionescu},
  title = {{Discriminability-enforcing loss to improve representation learning}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
  doi = {10.1109/cvprw56347.2022.00292},
}
Figure