Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning

Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang, Xu Liu, Lingling Li

Introduction

In real-world machine learning applications, data distributions are rarely balanced. Most datasets exhibit a "long-tailed" distribution where a few "head" classes contain the majority of samples, while many "tail" classes are represented by only a few examples. While recent advances in Long-Tailed Learning (LTL)—such as loss reweighting and specialized data augmentation—have significantly improved classification accuracy for these tail classes, a critical vulnerability remains: Robustness Imbalance.

This paper identifies that models trained on long-tailed data are fundamentally more fragile in their tail-class predictions. Even when a model achieves reasonable accuracy on clean tail-class data, its performance collapses disproportionately faster than head classes when faced with noise or interference. To address this, the authors introduce Orthogonal Uncertainty Representation (OUR), a plug-and-play method designed to stabilize the decision boundaries of tail classes by modeling uncertainty along the noisy, orthogonal directions of the data manifold.

The Problem: Robustness Imbalance

The authors argue that existing LTL methods focus too heavily on the clean data distribution and overlook how models behave as they move away from the "data manifold"—the low-dimensional space where natural, semantically clear images reside.

To quantify this, they introduce the Robustness Imbalance Factor (RIF). Their analysis reveals that as noise increases, the bias toward head classes is amplified. This suggests that the model's understanding of tail classes is "thin" and highly sensitive to any perturbation that pushes a sample off the learned manifold. Traditional augmentations often fail to solve this because they may accidentally move samples along the manifold, changing their semantic identity (e.g., morphing a '5' into a '3') and creating label ambiguity.

Proposed Method: Orthogonal Uncertainty Representation (OUR)

The OUR method is built on the principle that moving a sample in a direction orthogonal to the data manifold represents pure noise rather than a change in class identity. By augmenting features specifically in these directions, the model can learn to be invariant to noise without risking semantic confusion.

1. Defining the Noisy Manifold

The authors identify the orthogonal direction $U$ as the eigenvector corresponding to the smallest eigenvalue of the feature covariance matrix. This direction represents the axis where the data is thinnest and most susceptible to interference.

2. Feature Smearing

During training, OUR "smears" the feature embeddings of tail classes along this orthogonal direction. By adding controlled Gaussian noise specifically along $U$, the method forces the model to expand its decision regions for tail classes. This ensures that even if a test sample is noisy, it is more likely to remain within the correct class boundary.

3. Efficient End-to-End Training

Calculating the global covariance matrix for a manifold typically requires a full pass over the dataset, which is computationally prohibitive. The authors circumvent this by developing a strategy to approximate the covariance matrix using batch-wise updates from the previous epoch. This allows OUR to be integrated into standard training pipelines with negligible computational overhead and zero impact on inference speed.

Experimental Results

The method was rigorously evaluated on four standard long-tailed benchmarks: CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and the naturally long-tailed iNaturalist 2018.

Performance Gains

OUR demonstrates universal compatibility, acting as a performance booster for various base models and LTL algorithms:

  • Baseline Improvement: When added to standard Cross-Entropy (CE) loss on CIFAR-100-LT (IF 100), OUR improved Top-1 accuracy by 1.5%.
  • State-of-the-Art Integration: When combined with advanced methods like MiSLAS or ResLT, it provided consistent gains of 1% to 2%.
  • Tail-Class Impact: The most dramatic improvements occurred in the tail. On ImageNet-LT, OUR nearly doubled the tail-class performance of the baseline CE model, raising it from 7.7% to 14.5%.

Robustness and Reliability

Beyond raw accuracy, the RIF analysis confirmed that OUR significantly flattens the robustness curve. By reducing the fragility of tail-class representations, the resulting models are much more reliable when deployed in real-world environments where input noise is common.

Conclusion

Orthogonal Uncertainty Representation (OUR) provides a novel perspective on the long-tailed learning problem by shifting the focus from simple accuracy to manifold robustness. By intelligently augmenting data along orthogonal noise directions, OUR mitigates the "Bias Amplification" effect and creates more stable models.

While the robustness imbalance is not entirely eliminated, this work sets a new benchmark for reliable LTL. Future research may extend this manifold-based approach to non-linear feature spaces or apply similar uncertainty representations to domains like natural language processing and adversarial defense.

Summary updated

BibTeX
@inproceedings{ma2023orthogonal,
  author = {Yanbiao Ma and Licheng Jiao and Fang Liu and Shuyuan Yang and Xu Liu and Lingling Li},
  title = {{Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611698},
}
Figure