TL;DR
This paper introduces Frequency-Masked Embedding Inference (FEI), a novel non-contrastive method for self-supervised time series representation learning. By inferring relationships between frequency-masked time series segments in an embedding space, FEI eliminates the need for traditional positive/negative sample pairs, leading to improved generalization and state-of-the-art performance on various downstream tasks.
What Problem Does This Paper Address?
Self-supervised learning has revolutionized representation learning in fields like computer vision and NLP, but its application to time series data has lagged due to challenges in generalization. Current state-of-the-art methods often rely on contrastive learning, which requires carefully constructing positive and negative sample pairs. However, the continuous nature of time series makes defining these pairs difficult, leading to issues like the inability to capture subtle semantic differences and potential biases. This paper aims to overcome these limitations by proposing a fundamentally different, non-contrastive approach.
Methods
FEI introduces a novel "Frequency-Masked Embedding Inference" (FEI) framework. The core idea is to leverage a prompting strategy within an embedding space to learn continuous semantic relationships. It employs two inference branches:
- Target Embedding Inference: The model infers the embedding of a target time series (created by masking frequency components of the original series) using a "mask embedding" as a prompt.
- Mask Embedding Inference: Conversely, the model infers the "mask embedding" using the target time series embedding as a prompt.
This dual inference process, powered by a standard encoder and a momentum encoder to prevent representation collapse, allows the model to learn robust representations without explicit positive/negative sample pairs. The frequency masking is applied in the Fourier domain, allowing for fine-grained control over which frequency components are masked.
Datasets
The paper utilizes a variety of datasets for pre-training and downstream evaluation:
- Pre-training: SleepEEG (100 Hz)
- Classification Downstream Tasks: Gesture (100 Hz), FD-B (64 kHz), EMG (4 kHz), EPI (174 Hz), HAR (50 Hz), 128 UCR (various frequencies)
- Regression Downstream Tasks: C-MAPSS (1 Hz), Bearing (25.6 kHz)
These datasets cover a range of sampling frequencies and application domains, allowing for a comprehensive evaluation of the model's generalization capabilities.
Key Insights and Ideas
- Moving beyond Contrastive Learning: The paper demonstrates that effective time series representation learning can be achieved without the complexities and limitations of contrastive learning's positive/negative sample pairing.
- Continuous Semantic Modeling: By operating in the embedding space and inferring relationships between frequency-masked segments, FEI models continuous semantic variations in time series, which is crucial for their inherent nature.
- Frequency Domain Masking for Richer Semantics: Applying masks in the frequency domain allows for more nuanced control over information loss and semantic shifts compared to simple time-domain masking.
- Prompting for Inference: The use of mask embeddings as prompts for target embedding inference (and vice-versa) is a novel way to guide the learning process and encourage robust representations.
Key Results
FEI consistently outperforms existing contrastive-based methods across multiple benchmarks:
- Classification: FEI achieves an average accuracy improvement of 2.15% over existing methods in linear evaluation and approximately 3.50% in end-to-end fine-tuning on small-sample datasets.
- Regression: On the C-MAPSS and Bearing datasets, FEI shows superior performance with lower Mean Squared Error (MSE) and Mean Absolute Error (MAE) compared to baselines. For instance, on C-MAPSS, FEI's MSE is 9.31% lower than the next best method (TimesURL).
- Generalization: The paper highlights FEI's strong generalization ability, particularly on datasets with significant semantic differences from the pre-training data, a common weakness for time-domain masking methods.
Limitations and Future Work
- Masking Strategy Sensitivity: While FEI uses Discrete Frequency Masking (DFM), the paper notes that other masking strategies (Continuous Frequency Masking, Time-domain Masking) were explored. The performance can be sensitive to the choice of masking strategy, with DFM showing better generalization across varying frequencies. Future work could explore more adaptive or learned masking strategies.
- Hyperparameter Tuning: While the paper aims for minimal hyperparameter adjustments for downstream tasks, the pre-training setup still involves parameters like the masking ratio range (β₁ and β₂). Sensitivity analysis on these parameters is provided, suggesting optimal ranges, but further exploration could be beneficial.
- Computational Cost: The paper does not explicitly detail the computational cost of FEI compared to baselines, which is a common consideration in representation learning.
The authors suggest that their work provides new insights into non-contrastive approaches for time series representation learning, opening avenues for further research in this direction.
Summary updated