Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning

Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang, Xu Liu, Lingling Li

Introduction

In real-world machine learning applications, data distributions are rarely balanced. Most datasets exhibit a "long-tailed" distribution where a few "head" classes contain the majority of samples, while many "tail" classes are represented by only a few examples. While recent advances in Long-Tailed Learning (LTL)—such as loss reweighting and specialized data augmentation—have significantly improved classification accuracy for these tail classes, a critical vulnerability remains: Robustness Imbalance.

This paper identifies that models trained on long-tailed data are fundamentally more fragile in their tail-class predictions. Even when a model achieves reasonable accuracy on clean tail-class data, its performance collapses disproportionately faster than head classes when faced with noise or interference. To address this, the authors introduce Orthogonal Uncertainty Representation (OUR), a plug-and-play method designed to stabilize the decision boundaries of tail classes by modeling uncertainty along the noisy, orthogonal directions of the data manifold.

The Problem: Robustness Imbalance

The authors argue that existing LTL methods focus too heavily on the clean data distribution and overlook how models behave as they move away from the "data manifold"—the low-dimensional space where natural, semantically clear images reside.

To quantify this, they introduce the Robustness Imbalance Factor (RIF). Their analysis reveals that as noise increases, the bias toward head classes is amplified. This suggests that the model's understanding of tail classes is "thin" and highly sensitive to any perturbation that pushes a sample off the learned manifold. Traditional augmentations often fail to solve this because they may accidentally move samples along the manifold, changing their semantic identity (e.g., morphing a '5' into a '3') and creating label ambiguity.

Proposed Method: Orthogonal Uncertainty Representation (OUR)

The OUR method is built on the principle that moving a sample in a direction orthogonal to the data manifold represents pure noise rather than a change in class identity. By augmenting features specifically in these directions, the model can learn to be invariant to noise without risking semantic confusion.

1. Defining the Noisy Manifold

The authors identify the orthogonal direction $U$ as the eigenvector corresponding to the smallest eigenvalue of the feature covariance matrix. This direction represents the axis where the data is thinnest and most susceptible to interference.

2. Feature Smearing

During training, OUR "smears" the feature embeddings of tail classes along this orthogonal direction. By adding controlled Gaussian noise specifically along $U$, the method forces the model to expand its decision regions for tail classes. This ensures that even if a test sample is noisy, it is more likely to remain within the correct class boundary.

3. Efficient End-to-End Training

Calculating the global covariance matrix for a manifold typically requires a full pass over the dataset, which is computationally prohibitive. The authors circumvent this by developing a strategy to approximate the covariance matrix using batch-wise updates from the previous epoch. This allows OUR to be integrated into standard training pipelines with negligible computational overhead and zero impact on inference speed.

Experimental Results

The method was rigorously evaluated on four standard long-tailed benchmarks: CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and the naturally long-tailed iNaturalist 2018.

Performance Gains

OUR demonstrates universal compatibility, acting as a performance booster for various base models and LTL algorithms:

  • Baseline Improvement: When added to standard Cross-Entropy (CE) loss on CIFAR-100-LT (IF 100), OUR improved Top-1 accuracy by 1.5%.
  • State-of-the-Art Integration: When combined with advanced methods like MiSLAS or ResLT, it provided consistent gains of 1% to 2%.
  • Tail-Class Impact: The most dramatic improvements occurred in the tail. On ImageNet-LT, OUR nearly doubled the tail-class performance of the baseline CE model, raising it from 7.7% to 14.5%.

Robustness and Reliability

Beyond raw accuracy, the RIF analysis confirmed that OUR significantly flattens the robustness curve. By reducing the fragility of tail-class representations, the resulting models are much more reliable when deployed in real-world environments where input noise is common.

Conclusion

Orthogonal Uncertainty Representation (OUR) provides a novel perspective on the long-tailed learning problem by shifting the focus from simple accuracy to manifold robustness. By intelligently augmenting data along orthogonal noise directions, OUR mitigates the "Bias Amplification" effect and creates more stable models.

While the robustness imbalance is not entirely eliminated, this work sets a new benchmark for reliable LTL. Future research may extend this manifold-based approach to non-linear feature spaces or apply similar uncertainty representations to domains like natural language processing and adversarial defense.

Summary updated

Abstract

In scenarios with long-tailed distributions, the model's ability to identify tail classes is limited due to the under-representation of tail samples. Class rebalancing, information augmentation, and other techniques have been proposed to facilitate models to learn the potential distribution of tail classes. The disadvantage is that these methods generally pursue models with balanced class accuracy on the data manifold, while ignoring the ability of the model to resist interference. By constructing noisy data manifold, we found that the robustness of models trained on unbalanced data has a long-tail phenomenon. That is, even if the class accuracy is balanced on the data domain, it still has bias on the noisy data manifold. However, existing methods cannot effectively mitigate the above phenomenon, which makes the model vulnerable in long-tailed scenarios. In this work, we propose an Orthogonal Uncertainty Representation (OUR) of feature embedding and an end-to-end training strategy to improve the long-tail phenomenon of model robustness. As a general enhancement tool, OUR has excellent compatibility with other methods and does not require additional data generation, ensuring fast and efficient training. Comprehensive evaluations on long-tailed datasets show that our method significantly improves the long-tail phenomenon of robustness, bringing consistent performance gains to other long-tailed learning methods.
BibTeX
@inproceedings{ma2023orthogonal,
  author = {Yanbiao Ma and Licheng Jiao and Fang Liu and Shuyuan Yang and Xu Liu and Lingling Li},
  title = {{Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611698},
}
Figure