Introduction
In Task Incremental Learning (TIL), models learn a sequence of tasks over time. Most existing methods operate under a "closed-world" assumption, assuming all test data belongs to previously learned tasks. However, real-world deployments—such as autonomous vehicles or medical diagnostics—inevitably encounter "out-of-distribution" (OOD) samples that do not belong to any known task.
Current OOD detection methods, like Maximum Softmax Probability (MSP) or Energy scores, suffer from three primary flaws: high sensitivity to manually tuned thresholds, heavy dependency on model output quality (logits), and an inability to provide specific task identity for in-distribution samples.
The H2ST Framework
To address these challenges, the authors propose H2ST (Hierarchical Two-Sample Tests). This framework shifts OOD detection from simple scoring to a rigorous statistical hypothesis testing approach.
Key Components
- Two-Sample Testing: H2ST utilizes a Classifier Two-Sample Test (C2ST) to compare batches of new "target" samples against "source" samples stored in a memory buffer to determine if they share a distribution.
- Hierarchical Architecture: Tests are organized in a cascade. A sample enters at Layer 1; if it matches Task 1, the model performs an "early exit." If not, it proceeds through subsequent layers. Samples failing all $T$ layers are officially labeled OOD.
- Feature-Level Analysis: Tests are performed on internal feature maps rather than raw pixels, leveraging the model's learned high-level semantic representations.
- Threshold-Free Detection: By employing the Clopper-Pearson (CP) interval, H2ST makes decisions based on statistical confidence levels, eliminating the need for arbitrary numerical thresholds.
Experimental Evaluation
H2ST was evaluated across diverse benchmarks, including MNIST, CIFAR-10/100, Mini-ImageNet, and real-world streaming datasets like CoRe50 and Stream-51.
Performance and Efficiency
H2ST demonstrates a significant performance advantage over traditional baselines such as MSP, ODIN, and Energy-based models.
- OOD Detection & Task-ID: On CIFAR-10, H2ST achieved an average F1 score of ~92%, representing a 25.9% improvement over the previous state-of-the-art (MORE). Task identification accuracy (TA) also saw improvements exceeding 30%.
- Decoupling Detection from Classification: By using feature maps, H2ST can identify a sample as "known" even if the final classification layer is uncertain between similar classes.
- Computational Efficiency: The "early exit" strategy reduces overhead. In a 9-task setup, H2ST processed inputs in 16.7ms, compared to 22.0ms for a standard non-hierarchical C2ST.
Discussion and Future Work
While H2ST offers robust, threshold-free detection, it currently relies on a memory buffer to store samples from previous tasks. This dependency may limit its use in exemplar-free continual learning scenarios. Additionally, while the hierarchy is efficient for in-distribution data, OOD samples must traverse the entire cascade, which could introduce latency in systems with a very high number of tasks. Future research may explore adapting H2ST for Class Incremental Learning (CIL), where task boundaries are less distinct.
Summary updated