Temperature in Cosine-based Softmax Loss

Takumi KobayashiICCV 2025

TL;DR

This paper tackles the challenge of automatically tuning the "temperature" parameter in cosine-based softmax loss, a crucial factor for effective feature embedding in deep learning. They introduce a novel least-squares approach to adaptively estimate this temperature per sample, leading to improved performance and a better understanding of how temperature controls feature characteristics.

What Problem Does This Paper Address?

Deep learning models often rely on softmax cross-entropy loss for image classification. A variant, the cosine-based softmax loss (cos-softmax), has shown promise for learning discriminative feature embeddings, particularly in areas like face recognition and self-supervised learning. However, cos-softmax loss has a critical parameter called "temperature" that significantly impacts performance. This temperature parameter is responsible for scaling the cosine similarities, which are bounded between -1 and 1. Unfortunately, there's little prior knowledge on how to set this temperature, leading to manual, ad-hoc tuning that is often suboptimal and task-dependent. This paper aims to bridge this gap by proposing a method to adaptively estimate the temperature for each sample during training, making the cos-softmax loss more robust and interpretable.

Methods

The authors propose a least-squares (LS) approach to adaptively estimate the temperature parameter, denoted by $\kappa$ (the reciprocal of temperature $\tau$). Their method is rooted in a geometrical interpretation of the cos-softmax loss, drawing inspiration from sparse-representation classification.

  1. Geometrical Formulation: They view the feature vector $\mathbf{x}$ as being projected onto a convex hull spanned by the classifier weight vectors $\{\mathbf{w}_c\}_{c=1}^C$. This projection problem is then formulated as a least-squares optimization.
  2. Least-Squares Optimization: By substituting the softmax function for coefficients in the sparse-representation formulation, they derive a least-squares objective. Minimizing this objective yields an optimal temperature $\kappa^*$ for each sample.
  3. Sample-wise Optimization: Crucially, the optimization of $\kappa$ is performed per sample, not end-to-end. This means that for each input image, a suitable temperature is found, and then this temperature is used to compute the loss for that specific sample. This is achieved through a simple greedy search over a candidate set of $\kappa$ values.
  4. Probabilistic Approach (for comparison): They also explore a probabilistic approach based on the von Mises-Fisher distribution and maximum likelihood estimation, but find it less feasible in high-dimensional spaces compared to their proposed geometric LS method.

Datasets

The paper evaluates their method on several standard image classification datasets:

  • ImageNet: Used for initial comparisons of optimization methods and for analyzing the effect of $\kappa$ on feature distribution.
  • Cifar-10/100: Used for analyzing feature diversity and performance across different $\kappa$ values.
  • Food-101: Used for evaluating transfer learning performance.
  • iNaturalist2018: Used for imbalanced classification tasks.
  • Places-LT: Used for imbalanced classification tasks.
  • CUB-200, Aircraft-100, SUN-397, DTD, Flower-102: Used for transfer learning experiments.
  • MS1M-RetinaFace: Used for face recognition tasks.

Key Insights and Ideas

  • Temperature as a Controllable Knob: The paper demonstrates that the temperature parameter $\kappa$ in cos-softmax loss is not just a hyperparameter but a powerful tool to control the characteristics of learned feature representations in an explainable way.
  • Smaller $\kappa$ for Specialization: Smaller values of $\kappa$ (around the optimized $\kappa^*$) lead to more compact intra-class feature representations, resulting in higher confidence in classification and better performance on tasks like MISS and OOD detection. This is because smaller $\kappa$ pushes features closer to their respective class centroids.
  • Larger $\kappa$ for Generalization: Larger values of $\kappa$ encourage more diverse feature representations, which are less tied to specific class centroids. This diversity proves beneficial for transfer learning, allowing models to generalize better to unseen tasks.
  • $\kappa^*$ as a Lower Bound: The adaptive $\kappa^*$ estimated by the LS approach appears to serve as a lower bound for feasible temperatures that yield competitive performance. This suggests that $\kappa^*$ represents a sweet spot for specialization.
  • "Cosifying" Pretrained Models: The paper shows that models pre-trained with standard softmax loss can be "cosified" using the cos-softmax loss with specific $\kappa$ values, enhancing their specialization or generalization capabilities.

Key Results

  • Improved Accuracy: The proposed LS approach for adaptive temperature estimation achieves competitive or superior accuracy compared to fixed $\kappa$ values and standard softmax loss across various datasets and tasks (e.g., Table 2 shows ImageNet accuracy of 77.27% with LS-optimized $\kappa^*$ vs. 77.12% for standard softmax).
  • Effective OOD and MISS Detection: Smaller $\kappa$ values (around $\kappa^*$) significantly improve the detection of misclassified (MISS) and out-of-distribution (OOD) samples, outperforming standard softmax loss (Figure 4).
  • Strong Transfer Learning Performance: Models pre-trained with larger $\kappa$ values (e.g., $\kappa=60$) show significantly improved performance in linear probe transfer learning tasks compared to models trained with standard softmax loss (Table 4a).
  • Robustness to Class Number: The LS approach adaptively estimates $\kappa$ effectively across a wide range of class numbers (from 10 to 93431), demonstrating its scalability.
  • Interpretability: The analysis clearly links $\kappa$ values to specific feature characteristics (intra-class compactness vs. diversity), providing a more interpretable understanding of the learned representations.

Limitations and Future Work

  • Computational Cost: While the sample-wise optimization is efficient, it adds a computational overhead compared to a fixed temperature. The paper mentions a simple greedy search, but more advanced optimization strategies could be explored.
  • "Cosifying" Limitations: While "cosifying" pre-trained models shows promise, the paper notes that achieving optimal generalization with larger $\kappa$ might be harder for models already biased towards specialization.
  • Future Work: The authors suggest exploring the synergy of their method with post-hoc calibration techniques and further investigating the "cosification" process for different model architectures and pre-training strategies. The potential for this adaptive temperature to improve other loss functions beyond cos-softmax could also be an avenue for future research.

Summary updated

BibTeX
@inproceedings{kobayashi2025temperature,
  author = {Takumi Kobayashi},
  title = {{Temperature in Cosine-based Softmax Loss}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure