Abstract
Continual learning (CL) in real-world scenarios often faces the dual challenge of catastrophic forgetting and extreme data imbalance, known as Long-Tailed Continual Learning (LTCL). Traditional methods struggle to represent rare "tail" classes with minimal samples. We present ViRN (Variational Inference and Redistribution via Neighbors), a framework that leverages pre-trained embedding spaces to reconstruct missing statistical information for tail classes. By combining Variational Inference (VI) with a novel distribution trilateration technique, ViRN achieves a significant performance leap, outperforming state-of-the-art methods by an average of 10.24% across multiple benchmarks.
Introduction
In practical applications, data distributions are rarely uniform. A voice assistant may encounter common keywords thousands of times ("head" classes) while rarely hearing critical but infrequent sounds like a window breaking ("tail" classes). Standard CL models suffer from catastrophic forgetting, where new tasks overwrite old knowledge, and long-tailed bias, where the model ignores rare classes in favor of frequent ones. While generative replay is a common solution, it fails when a class has only one or two samples, as the model cannot accurately estimate the underlying distribution. ViRN addresses this extreme sample scarcity by "borrowing" structural information from semantically similar head classes.
Methodology
ViRN operates within the embedding space of high-quality pre-trained models (e.g., Wav2Vec for audio, DINOv2 for images) and employs a two-stage approach:
1. Variational Inference (VI)
Instead of using static prototypes, ViRN utilizes a Variational Autoencoder (VAE) to model each class as a Gaussian distribution. This probabilistic approach captures the uncertainty and variance of a class even when data is sparse.
2. Distribution Trilateration (RD)
This is the core innovation of ViRN. Recognizing that semantically similar classes cluster together, the model performs the following:
- Neighbor Identification: It calculates the 2-Wasserstein distance between the estimated tail distribution and all known classes to identify the $K$ nearest "head" neighbors.
- Reconstruction: The tail class's mean and covariance are reconstructed by interpolating between these well-defined neighbors.
- Adaptive Fusion: The initial VAE estimate is fused with the trilaterated distribution using a weighting factor that favors the neighbors when samples are scarce and shifts toward observed data as more samples arrive.
Experimental Setup
ViRN was evaluated on six long-tailed benchmarks with an imbalance ratio of $\rho = 0.01$ (the smallest class has 1% of the data of the largest). The datasets include four acoustic benchmarks (Speechcommands-LT, AudioMNIST-LT, ESC50-LT, UrbanSound8K-LT) and two visual benchmarks (CIFAR-100-LT, TinyImageNet-LT). Each dataset was partitioned into five sequential tasks to simulate a continual learning environment.
Key Results
ViRN demonstrates superior performance across all tested domains. As shown in the results table, ViRN achieved an overall summed accuracy of 524.26, a substantial improvement over the previous state-of-the-art, LAE (462.82). Notably, ViRN's performance closely approaches the "Upper Bound" (538.52), which represents a model trained on balanced, non-sequential data.
Ablation studies highlight the necessity of the Redistribution (RD) component. For instance, removing RD led to a 44.96% drop in new-class accuracy on the ESC50-LT dataset. Furthermore, ViRN maintains high stability; in the Speechcommands task, it kept forgetting rates as low as 2.94%.
Discussion and Conclusion
The success of ViRN stems from its ability to treat class distributions as geometric entities within a semantic manifold. By using a generative classifier ($P(x|y)$) rather than a discriminative one, the model naturally decouples feature learning from class frequency.
However, limitations remain. ViRN is dependent on the quality of the pre-trained backbone; if the embedding space lacks semantic coherence for a specific domain, trilateration may be less effective. Additionally, the computational cost of calculating Wasserstein distances for every class may pose challenges for real-time edge applications. Future work will focus on optimizing these geometric calculations and exploring the framework's robustness across even more diverse data modalities.
Summary updated