A Walkthrough for the Principle of Logit Separation

Gil Keren, Sivan Sabato, Björn W. Schuller

Abstract

Modern deep learning models often face a computational bottleneck in extreme classification tasks due to the final softmax layer. This paper introduces Single Logit Classification (SLC) and the Principle of Logit Separation (PoLS) to enable efficient inference by calculating only the logit of interest. We demonstrate that PoLS-aligned loss functions allow for global logit consistency, enabling significant speedups and improved accuracy in large-scale classification.

Introduction: The Softmax Bottleneck

In extreme classification scenarios, such as face recognition or web-scale image search, neural networks must often distinguish between millions of categories. Standard architectures utilize a softmax layer, which necessitates the computation of logits for every class to determine the probability of a single input. As the number of classes grows, this final layer becomes a primary computational bottleneck, often dominating the total computation time once the class count exceeds approximately 16,000 ($2^{14}$).

Computational bottleneck of computing all logits vs. SLC.

In many practical applications, we only need to verify if an input belongs to a specific class of interest. The authors propose Single Logit Classification (SLC) to address this, where only the queried class's logit is computed at test time, bypassing the need to calculate the entire distribution.

The Principle of Logit Separation (PoLS)

To make SLC effective, the model must satisfy the Principle of Logit Separation (PoLS). A model adheres to PoLS if the score for a correct class is consistently higher than the scores for incorrect classes across the entire dataset, rather than just within a single example.

Standard Cross-Entropy (CE) and Max-Margin losses fail to satisfy PoLS because they only enforce local separation—ensuring the correct class score is higher than others for a specific image. This leads to overlapping score distributions across the dataset, making it impossible to use a fixed threshold for single-logit classification.

Comparison of logit distributions between standard Cross-Entropy and PoLS-aligned loss.

In contrast, PoLS-aligned losses like Binary Cross-Entropy (BCE) or the newly introduced Batch Cross-Entropy enforce global separation. By treating classes as independent binary problems or applying KL-divergence over an entire batch, these methods ensure that a specific logit value reliably indicates the presence or absence of a class regardless of the "competition."

Experimental Results

The researchers evaluated PoLS-aligned models on benchmarks including MNIST, SVHN, CIFAR-10/100, and ImageNet.

Efficiency Gains

The speedup provided by SLC is most pronounced in large-scale settings. On ImageNet-scale models with $2^{18.5}$ classes, SLC achieved a 21x speedup for AlexNet and a 12.7x speedup for Inception-v3. For standard 10-class or 100-class problems, the overhead of the rest of the network makes the SLC speedup negligible, but the scalability for extreme classification is clear.

Accuracy and Reliability

Using PoLS-aligned losses improved SLC performance metrics significantly. For instance, AUPRC improved by over 20% compared to non-aligned losses. Because these losses separate the distributions of true and false logits, a simple threshold can be used at inference time to achieve high precision and recall without needing to see the scores of other categories.

Conclusion

The Principle of Logit Separation provides a framework for scaling neural networks to massive classification tasks without linear increases in inference time. While the benefits are most significant for large class counts, the global consistency enforced by PoLS-aligned losses offers a more robust foundation for threshold-based decision-making. Future work includes extending these principles to regression tasks and real-time machine translation.

Summary updated

Abstract

We consider neural network training, in applications in which there are many possible classes, but at test-time, the task is a binary classification task of determining whether the given example belongs to a specific class. We define the Single Logit Classification (SLC) task: training the network so that at test-time, it would be possible to accurately identify whether the example belongs to a given class in a computationally efficient manner, based only on the output logit for this class. We propose a natural principle, the Principle of Logit Separation, as a guideline for choosing and designing loss functions that are suitable for SLC. We show that the Principle of Logit Separation is a crucial ingredient for success in the SLC task, and that SLC results in considerable speedups when the number of classes is large.
BibTeX
@inproceedings{keren2019walkthrough,
  author = {Gil Keren and Sivan Sabato and Björn W. Schuller},
  title = {{A Walkthrough for the Principle of Logit Separation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/861},
}

Loading related content...

Figure