Class-Level Logit Perturbation

Mengyang Li, Fengguang Su, Ou Wu, Ji Zhang

Abstract

This paper introduces Class-Level Logit Perturbation (LPL), a novel method that explicitly perturbs the raw output scores (logits) of a neural network to improve model generalization. By strategically increasing or decreasing training loss through optimized perturbations, LPL serves as a versatile "plug-in" for challenging tasks such as long-tail (imbalanced) and multi-label classification. Theoretical analysis demonstrates that increasing training loss via logit perturbation is equivalent to positive data augmentation, providing a principled framework for this approach.

Introduction

In deep learning, data typically progresses through three stages: features, logits, and labels. While perturbations at the feature level (e.g., image flipping) and label level (e.g., label smoothing) are common, the intermediate logit stage has been largely overlooked for explicit optimization. Existing methods for handling imbalanced data often modify logits implicitly, but they lack a unified framework. As shown in Figure 2, methods like Logit Adjustment (LA) and LDAM exhibit specific patterns of relative loss variation across head and tail categories, suggesting that controlled logit perturbation could be a powerful tool for balancing model focus.

Relative loss variation across category IDs for existing methods like ISDA, LA, and LDAM.

Methodology

The core of the proposed approach is the Logit Perturbation Method (LPL). It treats logit perturbation as a form of implicit data augmentation, categorized into two types:

  1. Positive Augmentation: Perturbing logits to increase training loss (similar to adversarial training) forces the model to learn harder features, effectively augmenting the presence of minority classes.
  2. Negative Augmentation: Perturbing logits to decrease training loss restrains the model's focus on dominant "head" classes in imbalanced datasets.

LPL utilizes a PGD-like optimization process to iteratively find the optimal perturbation ($\delta$) that maximizes or minimizes loss according to the class's needs. The framework is designed as a plug-in, requiring no changes to the network architecture; the standard Cross-Entropy loss is simply replaced with the LPL loss during training.

Schematic of the LPL framework showing category splitting and loss optimization.

Experimental Results

LPL was evaluated across standard classification (CIFAR-10/100), long-tail classification (CIFAR-LT, iNaturalist), and multi-label classification (MS-COCO) benchmarks. Key findings include:

  • Long-tail Learning: On CIFAR-100-LT (100:1 imbalance), LPL achieved a 55.75% error rate, surpassing the previous state-of-the-art Logit Adjustment (56.11%). On the large-scale iNaturalist 2018 dataset, LPL reached a top-1 error of 30.59%, significantly improving over standard baselines.
  • Multi-label Classification: On MS-COCO, LPL achieved a mean Average Precision (mAP) of 85.4%, outperforming other logit-based methods like NTR and LC.
  • General Augmentation: On balanced CIFAR-100 with ResNet-110, LPL achieved a top-1 error of 25.87%, notably better than the popular ISDA method (27.57%).

Conclusion

Class-Level Logit Perturbation (LPL) provides a theoretically grounded and practically efficient method for improving deep learning models. By explicitly controlling training loss through logit-level perturbations, it offers a unified solution for data imbalance and multi-label challenges. Future work may explore extending LPL to domains like Natural Language Processing and automating hyperparameter selection through meta-learning.

Summary updated

Abstract

Features, logits, and labels are the three primary data when a sample passes through a deep neural network (DNN). Feature perturbation and label perturbation receive increasing attention in recent years. They have been proven to be useful in various deep learning approaches. For example, (adversarial) feature perturbation can improve the robustness or even generalization capability of learned models. However, limited studies have explicitly explored for the perturbation of logit vectors. This work discusses several existing methods related to class-level logit perturbation. A unified viewpoint between regular/irregular data augmentation and loss variations incurred by logit perturbation is established. A theoretical analysis is provided to illuminate why class-level logit perturbation is useful. Accordingly, new methodologies are proposed to explicitly learn to perturb logits for both the single-label and multilabel classification tasks. Meta-learning is also leveraged to determine the regular or irregular augmentation for each class. Extensive experiments on benchmark image classification datasets and their long-tail versions indicated the competitive performance of our learning method. As it only perturbs on logit, it can be used as a plug-in to fuse with any existing classification algorithms. All the codes are available at https://github.com/limengyang1992/lpl.
BibTeX
@article{li2024class,
  author = {Mengyang Li and Fengguang Su and Ou Wu and Ji Zhang},
  title = {{Class-Level Logit Perturbation}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  url = {https://openreview.net/forum?id=vjAjJWQNRe},
}
Figure