Contrastive-center loss for deep neural networks

Ce Qi, Fei Su

TL;DR

Contrastive-center loss is a new auxiliary training signal that improves how deep neural networks group similar data and separate different classes. By optimizing a ratio that minimizes distance to a sample's own class center while maximizing distance to all other class centers, it achieves superior discriminative power without the complex data-sampling requirements of triplet or contrastive losses.

What Problem Does This Paper Address?

In deep learning for classification and face recognition, the standard Softmax loss is excellent at separating classes, but it doesn't explicitly encourage "compactness." This means features belonging to the same class can be spread out widely in the feature space, making the model less robust.

Previous attempts to fix this include:

  • Contrastive and Triplet Losses: These work well but require complex "mining" strategies to find pairs or triplets of images to compare, which is computationally expensive ($O(N^2)$ or $O(N^3)$ complexity).
  • Center Loss: This was a breakthrough that learned a "center" for each class and pulled samples toward it. However, it only focused on intra-class compactness (pulling things together) and ignored inter-class separability (pushing different groups apart). If class centers start too close together, Center Loss doesn't do much to move them.

Methods

The authors propose Contrastive-center loss ($L_{ct-c}$). Unlike Center Loss, which only looks at the distance to the correct center, this new loss function is a ratio:

  1. The Numerator: The squared distance between a sample and its corresponding class center (encouraging compactness).
  2. The Denominator: The sum of squared distances between that same sample and all other class centers (encouraging separability).

By minimizing this ratio, the network is forced to pull samples toward their own center while simultaneously pushing them away from the "territory" of other classes.

Crucially, this is used as an auxiliary loss. The total loss function is $L = L_{softmax} + \lambda L_{ct-c}$, where $\lambda$ is a hyperparameter that controls how much weight to give to the feature distribution versus the raw classification accuracy. The class centers are updated during training using a mini-batch approach, similar to how network weights are updated via backpropagation.

Datasets

The authors evaluated their method on three standard benchmarks:

  • MNIST: Hand-written digits (60k training, 10k testing). Used primarily for visualization and baseline accuracy.
  • CIFAR-10: 10 classes of natural images (50k training, 10k testing).
  • LFW (Labeled Faces in the Wild): 13,233 face images of 5,749 people. For this, they pre-trained on the CASIA-WebFace dataset (490k images).

Key Insights and Ideas

  • The Power of the Ratio: By using a ratio instead of a simple sum, the loss function becomes much more sensitive to the relative geometry of the feature space. It prevents the "lazy" convergence seen in standard Center Loss where classes might remain bunched together.
  • No Mining Required: Unlike Triplet Loss, which needs careful selection of "hard" examples to work, Contrastive-center loss uses all class centers in every calculation. This makes the training process simpler and more stable.
  • Geometric Intuition: Visualization of the feature space (see Fig. 1 in the paper) shows that while Softmax makes classes "separable," Contrastive-center loss makes them "clusters." The average distance between class centers increased by 3.3 to 5 times compared to standard Center Loss.

Key Results

The proposed loss consistently outperformed both standard Softmax and the original Center Loss:

  • MNIST: Achieved 99.17% accuracy, compared to 98.8% (Softmax) and 98.94% (Center Loss).
  • CIFAR-10: Using a 20-layer ResNet, it reached 92.45% accuracy, a 1.2% improvement over Softmax alone.
  • LFW (Face Recognition): Achieved a verification accuracy of 98.68%. When compared to a model trained with standard Center Loss on the same architecture, it provided a 0.13% boost, proving its effectiveness in high-dimensional recognition tasks.

Limitations and Future Work

  • Hyperparameter Sensitivity: The model introduces $\lambda$ (loss weight) and $\alpha$ (center learning rate). Finding the optimal balance for these can require extra tuning.
  • Scalability of Centers: While more efficient than triplet mining, the model still requires maintaining and updating $k$ class centers in memory. For datasets with millions of classes (like massive-scale face recognition), this could become a memory bottleneck.
  • Fixed Centers: The paper doesn't deeply explore how the initialization of centers affects the final geometry, though they note that the contrastive term helps mitigate poor initializations.

Summary updated

Abstract

The deep convolutional neural network(CNN) has significantly raised the performance of image classification and face recognition. Softmax is usually used as supervision, but it only penalizes the classification loss. In this paper, we propose a novel auxiliary supervision signal called contrastive-center loss, which can further enhance the discriminative power of the features, for it learns a class center for each class. The proposed contrastive-center loss simultaneously considers intra-class compactness and inter-class separability, by penalizing the contrastive values between: (1)the distances of training samples to their corresponding class centers, and (2)the sum of the distances of training samples to their non-corresponding class centers. Experiments on different datasets demonstrate the effectiveness of contrastive-center loss.
BibTeX
@inproceedings{qi2017contrastive,
  author = {Ce Qi and Fei Su},
  title = {{Contrastive-center loss for deep neural networks}},
  booktitle = {ICIP},
  year = {2017},
  url = {https://openreview.net/forum?id=gIa9Icncsr},
}

Loading related content...

Figure