Abstract
Modern deep learning models often face a computational bottleneck in extreme classification tasks due to the final softmax layer. This paper introduces Single Logit Classification (SLC) and the Principle of Logit Separation (PoLS) to enable efficient inference by calculating only the logit of interest. We demonstrate that PoLS-aligned loss functions allow for global logit consistency, enabling significant speedups and improved accuracy in large-scale classification.
Introduction: The Softmax Bottleneck
In extreme classification scenarios, such as face recognition or web-scale image search, neural networks must often distinguish between millions of categories. Standard architectures utilize a softmax layer, which necessitates the computation of logits for every class to determine the probability of a single input. As the number of classes grows, this final layer becomes a primary computational bottleneck, often dominating the total computation time once the class count exceeds approximately 16,000 ($2^{14}$).
In many practical applications, we only need to verify if an input belongs to a specific class of interest. The authors propose Single Logit Classification (SLC) to address this, where only the queried class's logit is computed at test time, bypassing the need to calculate the entire distribution.
The Principle of Logit Separation (PoLS)
To make SLC effective, the model must satisfy the Principle of Logit Separation (PoLS). A model adheres to PoLS if the score for a correct class is consistently higher than the scores for incorrect classes across the entire dataset, rather than just within a single example.
Standard Cross-Entropy (CE) and Max-Margin losses fail to satisfy PoLS because they only enforce local separation—ensuring the correct class score is higher than others for a specific image. This leads to overlapping score distributions across the dataset, making it impossible to use a fixed threshold for single-logit classification.
In contrast, PoLS-aligned losses like Binary Cross-Entropy (BCE) or the newly introduced Batch Cross-Entropy enforce global separation. By treating classes as independent binary problems or applying KL-divergence over an entire batch, these methods ensure that a specific logit value reliably indicates the presence or absence of a class regardless of the "competition."
Experimental Results
The researchers evaluated PoLS-aligned models on benchmarks including MNIST, SVHN, CIFAR-10/100, and ImageNet.
Efficiency Gains
The speedup provided by SLC is most pronounced in large-scale settings. On ImageNet-scale models with $2^{18.5}$ classes, SLC achieved a 21x speedup for AlexNet and a 12.7x speedup for Inception-v3. For standard 10-class or 100-class problems, the overhead of the rest of the network makes the SLC speedup negligible, but the scalability for extreme classification is clear.
Accuracy and Reliability
Using PoLS-aligned losses improved SLC performance metrics significantly. For instance, AUPRC improved by over 20% compared to non-aligned losses. Because these losses separate the distributions of true and false logits, a simple threshold can be used at inference time to achieve high precision and recall without needing to see the scores of other categories.
Conclusion
The Principle of Logit Separation provides a framework for scaling neural networks to massive classification tasks without linear increases in inference time. While the benefits are most significant for large class counts, the global consistency enforced by PoLS-aligned losses offers a more robust foundation for threshold-based decision-making. Future work includes extending these principles to regression tasks and real-time machine translation.
Summary updated