Abstract
rETF-semiSL is a novel semi-supervised learning framework designed to enhance time-series classification by inducing "Neural Collapse" (NC) during pre-training. By employing a fixed geometric classifier (rETF) and a unique "forward mixing" augmentation strategy, the framework aligns class embeddings into a symmetric, equidistant Simplex Equiangular Tight Frame (ETF). Experimental results across multiple architectures, including LSTMs and Transformers, demonstrate a 12% average performance boost, proving superior transferability and computational efficiency compared to standard self-supervised methods.
Introduction
Deep learning models for time series, such as those used in medical diagnostics or activity tracking, are notoriously data-hungry. However, labeling this data is often expensive and time-consuming. While Self-Supervised Learning (SSL) has emerged as a solution—pre-training models on unlabeled data using heuristic "pretext" tasks—these tasks often fail to prepare the model's internal geometry for the final classification goal. This creates a "transferability gap" where learned features are not optimal for downstream tasks.
The authors address this gap by leveraging Neural Collapse (NC). NC is a phenomenon where class embeddings converge to the vertices of a Simplex Equiangular Tight Frame (ETF)—the most symmetric and equidistant arrangement possible on a sphere. rETF-semiSL forces the model to adopt this optimal geometry early in the training process, creating classification-ready features.
Methodology
Geometric Priors and the rETF Classifier
At the heart of the framework is the rETF Classifier. Unlike standard learnable linear layers, this classifier uses a fixed ETF matrix as a geometric target. To allow for flexibility, the model learns a rotation matrix that aligns the feature space with this rigid target. This process is further refined by a Modified Center Loss function, which encourages embeddings to cluster tightly around class centers by optimizing both vector direction and distance.
Forward Mixing: A Temporal-Aware Augmentation
To handle unlabeled data effectively, the authors introduce Forward Mixing. Standard augmentations like Gaussian noise often break the temporal logic of a signal. Forward Mixing instead creates new samples by interpolating between a time step ($x_t$) and its successor ($x_{t+1}$). This "physics-aware" perturbation respects the underlying dynamics of the time series, helping the model learn invariance to small temporal shifts.
Semi-Supervised Loop
The framework utilizes an Alternating Pseudo-Labeling strategy. The model iteratively assigns labels to unlabeled samples based on its current state and then updates the encoder using a combination of ground-truth and pseudo-labels. Optionally, generative tasks like reconstruction can be integrated as auxiliary objectives to help the encoder ignore temporal jitter.
Experimental Results
The framework was evaluated on multivariate benchmarks from the UEA archive, including HAR (Human Activity Recognition), Epilepsy, and Heartbeat datasets. Testing spanned four backbone architectures: LSTMs, iTransformers, TimesNet, and Mamba (State-Space Models).
Key findings include:
- Performance Gains: rETF-semiSL achieved an average relative improvement of 12% in classification accuracy over standard supervised and contrastive methods (e.g., SimCLR, SupCon).
- Computational Efficiency: The method is significantly lighter than contrastive learning. While contrastive methods scale at $O(N^2)$, the rETF head scales at $O(NdC)$, where $C$ is the number of classes.
- Enhanced Transferability: Representation quality metrics like LogME and NLEEP confirmed that rETF-learned features are more transferable to final classification tasks than those learned via simple reconstruction.
Conclusion
rETF-semiSL demonstrates that enforcing geometric priors like Neural Collapse can significantly bridge the gap between pre-training and classification in time-series analysis. While joint training with generative tasks showed diminishing returns for complex models like Transformers, the core rETF and Forward Mixing components provide a robust, efficient path for high-performance semi-supervised learning.
Summary updated