Introduction
Time series data analysis is foundational to fields ranging from finance to healthcare. However, extracting robust representations remains challenging due to the inherent tension between capturing long-range global dependencies and sensitive local features. Existing methods often lean too heavily on one or the other, or fail to utilize available label information effectively. The Supervised COntrastive Temporal Transformer (SCOTT) addresses these hurdles by integrating the global reach of Transformers with the local precision of Temporal Convolutional Networks (TCNs) within a supervised contrastive learning framework.
The Problem: Global Trends vs. Local Anomalies
Time series data is notoriously difficult to represent effectively because it often contains both "global" trends (long-term patterns) and "local" anomalies (short-term spikes or dips). While self-supervised contrastive learning has been successful in computer vision, applying it to time series faces three major hurdles:
- Augmentation Sensitivity: Standard data augmentations (like adding noise) can destroy the inherent meaning of sensitive time series data, such as medical signals or stock prices.
- Architectural Trade-offs: Models usually focus on either global dependencies (Transformers) or local features (CNNs), but rarely both efficiently.
- Label Underutilization: Most contrastive methods are unsupervised; they do not leverage available labels to group similar instances together in the representation space.
The SCOTT Framework
The authors propose three key innovations within the SCOTT framework to bridge these gaps:
1. The Temporal-Transformer Encoder
Traditional Transformers use multi-head self-attention to model global context but rely on position-wise feed-forward networks (FFN) that treat each time step independently. SCOTT replaces these FFNs with dilated causal convolutions. This hybrid approach ensures that the model captures local interactions and temporal hierarchies within the same block that processes global dependencies. This "Temporal-Transformer" encoder acts as a powerful feature extractor that is both efficient and effective, providing a "best-of-both-worlds" solution.
2. Simplified Supervised Contrastive (SupCon) Loss
Contrastive learning has revolutionized computer vision, but its application to time series is often hindered by the computational cost of supervised variants. The authors mathematically prove that treating each data augmentation as an independent instance of its class is equivalent to the original, more complex SupCon loss. This simplification reduces computational overhead significantly, allowing the model to scale more effectively as training datasets grow while still creating tighter, more separable clusters in the representation space.
3. Tail-Focused Augmentation for Change Point Detection
For online Change Point Detection (CPD), standard augmentations can be counterproductive. SCOTT employs a "tail-focused" augmentation strategy for sliding windows, where only the older data (the "tail") is augmented. This forces the model to focus on the most recent, un-augmented data points to detect changes in real-time, which is critical for high-stakes monitoring tasks.
Experimental Evaluation
The model was rigorously tested across several domains, including the UCR Time Series Archive (45 diverse datasets) and the USC-HAD human activity dataset.
One of the most compelling applications was in the real-world surgical monitoring of Electrocochleography (ECochG) data. This specialized dataset tracks the inner ear's response to sound during cochlear implant surgeries to detect "traumatic drops" that might indicate surgical damage.
Key Results
- Top-Tier Classification: On the 45 UCR datasets, SCOTT achieved the highest average rank against nine baseline models, including heavyweights like Rocket and InceptionTime. Even a simple, "weak" classifier (a basic MLP) performed at state-of-the-art levels when fed SCOTT's representations.
- Surgical Precision: In the online Change Point Detection task for ECochG data, SCOTT achieved an AUPRC (Area Under Precision-Recall Curve) of ~98%, significantly outperforming existing Bayesian and Deep Learning baselines.
- Early Detection Potential: The model showed a strong ability to detect "traumatic drops" in surgical signals up to 30 steps before they fully manifested, highlighting its potential for real-time clinical decision support.
Conclusion and Future Work
SCOTT represents a significant step forward in time series representation learning. By harmonizing global and local feature extraction and simplifying supervised contrastive objectives, it provides a versatile tool for both general classification and high-stakes real-time monitoring.
While highly effective, the authors note that future work should focus on extending the Temporal-Transformer to handle complex, high-dimensional multivariate data more natively and moving from simulated "online" environments to live clinical deployments. The study also confirms that augmentation is domain-specific: while warping works best for image-based time series, jittering (adding noise) remains superior for sensor and audio data where background noise is expected.
Summary updated