Abstract
This paper introduces Class-Level Logit Perturbation (LPL), a novel method that explicitly perturbs the raw output scores (logits) of a neural network to improve model generalization. By strategically increasing or decreasing training loss through optimized perturbations, LPL serves as a versatile "plug-in" for challenging tasks such as long-tail (imbalanced) and multi-label classification. Theoretical analysis demonstrates that increasing training loss via logit perturbation is equivalent to positive data augmentation, providing a principled framework for this approach.
Introduction
In deep learning, data typically progresses through three stages: features, logits, and labels. While perturbations at the feature level (e.g., image flipping) and label level (e.g., label smoothing) are common, the intermediate logit stage has been largely overlooked for explicit optimization. Existing methods for handling imbalanced data often modify logits implicitly, but they lack a unified framework. As shown in Figure 2, methods like Logit Adjustment (LA) and LDAM exhibit specific patterns of relative loss variation across head and tail categories, suggesting that controlled logit perturbation could be a powerful tool for balancing model focus.
Methodology
The core of the proposed approach is the Logit Perturbation Method (LPL). It treats logit perturbation as a form of implicit data augmentation, categorized into two types:
- Positive Augmentation: Perturbing logits to increase training loss (similar to adversarial training) forces the model to learn harder features, effectively augmenting the presence of minority classes.
- Negative Augmentation: Perturbing logits to decrease training loss restrains the model's focus on dominant "head" classes in imbalanced datasets.
LPL utilizes a PGD-like optimization process to iteratively find the optimal perturbation ($\delta$) that maximizes or minimizes loss according to the class's needs. The framework is designed as a plug-in, requiring no changes to the network architecture; the standard Cross-Entropy loss is simply replaced with the LPL loss during training.
Experimental Results
LPL was evaluated across standard classification (CIFAR-10/100), long-tail classification (CIFAR-LT, iNaturalist), and multi-label classification (MS-COCO) benchmarks. Key findings include:
- Long-tail Learning: On CIFAR-100-LT (100:1 imbalance), LPL achieved a 55.75% error rate, surpassing the previous state-of-the-art Logit Adjustment (56.11%). On the large-scale iNaturalist 2018 dataset, LPL reached a top-1 error of 30.59%, significantly improving over standard baselines.
- Multi-label Classification: On MS-COCO, LPL achieved a mean Average Precision (mAP) of 85.4%, outperforming other logit-based methods like NTR and LC.
- General Augmentation: On balanced CIFAR-100 with ResNet-110, LPL achieved a top-1 error of 25.87%, notably better than the popular ISDA method (27.57%).
Conclusion
Class-Level Logit Perturbation (LPL) provides a theoretically grounded and practically efficient method for improving deep learning models. By explicitly controlling training loss through logit-level perturbations, it offers a unified solution for data imbalance and multi-label challenges. Future work may explore extending LPL to domains like Natural Language Processing and automating hyperparameter selection through meta-learning.
Summary updated