rETF-semiSL: Semi-Supervised Learning for Neural Collapse in Temporal Data

Yuhan Xie, William Cappelletti, Mahsa Shoaran, Pascal FrossardarXiv 2025 · DOI · arXiv

Abstract

rETF-semiSL is a novel semi-supervised learning framework designed to enhance time-series classification by inducing "Neural Collapse" (NC) during pre-training. By employing a fixed geometric classifier (rETF) and a unique "forward mixing" augmentation strategy, the framework aligns class embeddings into a symmetric, equidistant Simplex Equiangular Tight Frame (ETF). Experimental results across multiple architectures, including LSTMs and Transformers, demonstrate a 12% average performance boost, proving superior transferability and computational efficiency compared to standard self-supervised methods.

Introduction

Deep learning models for time series, such as those used in medical diagnostics or activity tracking, are notoriously data-hungry. However, labeling this data is often expensive and time-consuming. While Self-Supervised Learning (SSL) has emerged as a solution—pre-training models on unlabeled data using heuristic "pretext" tasks—these tasks often fail to prepare the model's internal geometry for the final classification goal. This creates a "transferability gap" where learned features are not optimal for downstream tasks.

The authors address this gap by leveraging Neural Collapse (NC). NC is a phenomenon where class embeddings converge to the vertices of a Simplex Equiangular Tight Frame (ETF)—the most symmetric and equidistant arrangement possible on a sphere. rETF-semiSL forces the model to adopt this optimal geometry early in the training process, creating classification-ready features.

Methodology

Geometric Priors and the rETF Classifier

At the heart of the framework is the rETF Classifier. Unlike standard learnable linear layers, this classifier uses a fixed ETF matrix as a geometric target. To allow for flexibility, the model learns a rotation matrix that aligns the feature space with this rigid target. This process is further refined by a Modified Center Loss function, which encourages embeddings to cluster tightly around class centers by optimizing both vector direction and distance.

The rETF framework optimizes the feature space to align with a fixed geometric target (ETF) using rotation and center loss.

Forward Mixing: A Temporal-Aware Augmentation

To handle unlabeled data effectively, the authors introduce Forward Mixing. Standard augmentations like Gaussian noise often break the temporal logic of a signal. Forward Mixing instead creates new samples by interpolating between a time step ($x_t$) and its successor ($x_{t+1}$). This "physics-aware" perturbation respects the underlying dynamics of the time series, helping the model learn invariance to small temporal shifts.

Forward Mixing augmentation creates new samples by interpolating between adjacent time steps, preserving temporal dynamics.

Semi-Supervised Loop

The framework utilizes an Alternating Pseudo-Labeling strategy. The model iteratively assigns labels to unlabeled samples based on its current state and then updates the encoder using a combination of ground-truth and pseudo-labels. Optionally, generative tasks like reconstruction can be integrated as auxiliary objectives to help the encoder ignore temporal jitter.

Experimental Results

The framework was evaluated on multivariate benchmarks from the UEA archive, including HAR (Human Activity Recognition), Epilepsy, and Heartbeat datasets. Testing spanned four backbone architectures: LSTMs, iTransformers, TimesNet, and Mamba (State-Space Models).

Key findings include:

  • Performance Gains: rETF-semiSL achieved an average relative improvement of 12% in classification accuracy over standard supervised and contrastive methods (e.g., SimCLR, SupCon).
  • Computational Efficiency: The method is significantly lighter than contrastive learning. While contrastive methods scale at $O(N^2)$, the rETF head scales at $O(NdC)$, where $C$ is the number of classes.
  • Enhanced Transferability: Representation quality metrics like LogME and NLEEP confirmed that rETF-learned features are more transferable to final classification tasks than those learned via simple reconstruction.

Conclusion

rETF-semiSL demonstrates that enforcing geometric priors like Neural Collapse can significantly bridge the gap between pre-training and classification in time-series analysis. While joint training with generative tasks showed diminishing returns for complex models like Transformers, the core rETF and Forward Mixing components provide a robust, efficient path for high-performance semi-supervised learning.

Summary updated

Abstract

Deep neural networks for time series must capture complex temporal patterns, to effectively represent dynamic data. Self- and semi-supervised learning methods show promising results in pre-training large models, which -- when finetuned for classification -- often outperform their counterparts trained from scratch. Still, the choice of pretext training tasks is often heuristic and their transferability to downstream classification is not granted, thus we propose a novel semi-supervised pre-training strategy to enforce latent representations that satisfy the Neural Collapse phenomenon observed in optimally trained neural classifiers. We use a rotational equiangular tight frame-classifier and pseudo-labeling to pre-train deep encoders with few labeled samples. Furthermore, to effectively capture temporal dynamics while enforcing embedding separability, we integrate generative pretext tasks with our method, and we define a novel sequential augmentation strategy. We show that our method significantly outperforms previous pretext tasks when applied to LSTMs, transformers, and state-space models on three multivariate time series classification datasets. These results highlight the benefit of aligning pre-training objectives with theoretically grounded embedding geometry.
BibTeX
@misc{xie2025retf,
  author = {Yuhan Xie and William Cappelletti and Mahsa Shoaran and Pascal Frossard},
  title = {{rETF-semiSL: Semi-Supervised Learning for Neural Collapse in Temporal Data}},
  year = {2025},
  eprint = {2508.10147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.10147},
}
Figure