While Transformer models have revolutionized natural language processing and computer vision, their application to Time Series Forecasting (TSF) remains somewhat of a "black box." We know these models achieve state-of-the-art results, but we lack a deep understanding of how they transform raw numerical sequences into accurate future predictions.
Unlike images or text, time series data lacks intuitive "natural" features that are easy for humans to visualize. To bridge this interpretability gap, recent research employs Manifold Learning. This approach relies on the "manifold hypothesis": the idea that high-dimensional data (such as a 512-dimension hidden layer in a neural network) actually resides on a much simpler, low-dimensional geometric surface. By measuring the properties of this surface, researchers can track how a model processes information layer by layer.
Measuring the Manifold: ID and Curvature
To analyze the internal states of state-of-the-art TSF Transformers like Autoformer and FEDformer, researchers utilize two specialized geometric metrics:
- Intrinsic Dimension (ID): Estimated using the Two Nearest Neighbors (TwoNN) method, ID reveals the minimum number of variables needed to describe the data without losing information.
- Mean Absolute Principal Curvature (MAPC): Calculated via Curvature Aware Manifold Learning (CAML), this metric measures how much the manifold "bends" or deviates from being a perfectly flat plane.
By extracting latent representations from every layer of these models across various large-scale datasets (including ETT, Electricity, Traffic, and Weather), researchers can plot "geometric profiles" that describe the model's internal state throughout the forecasting process.
The Two-Phase Geometric Journey
One of the most striking discoveries is how TSF Transformers process data compared to models in other domains. Image classification models typically show a "hunchback" ID profile, where dimensionality rises and then falls. In contrast, TSF Transformers exhibit a distinct two-phase journey.
As seen in the geometric profiles, during the encoding phase (layers 1-4), the manifold stays relatively flat or simplifies. However, during the decoding phase (layers 5-7), both dimensionality and curvature experience a sharp spike. This suggests that the decoder actively "expands" the representation to capture the complex variance required for multi-step future prediction.
Curvature as a Predictor of Performance
Perhaps the most actionable insight from this geometric analysis is the relationship between manifold curvature and model accuracy. There is a significant positive correlation between the curvature of the final layer and the model's Mean Squared Error (MSE).
Essentially, flatter manifolds lead to better forecasts. The correlation coefficient between the final layer's MAPC and the test MSE was found to be 0.76 for Autoformer and 0.70 for FEDformer. This is a major breakthrough because it suggests we could potentially evaluate—or even regularize—models based on their internal geometry without needing a holdout test set.
Dataset Fingerprints and Rapid Convergence
The geometric analysis also revealed that models learn these manifolds incredibly fast. The geometric profile of a model stabilizes within just the first five epochs of training. Furthermore, the encoder's geometry converges almost immediately (within 2 epochs), indicating that the "heavy lifting" of learning the specific forecasting task happens primarily in the decoder.
Additionally, the data itself leaves a distinct mark on the model's geometry. Related datasets produce remarkably similar geometric manifolds. For example, Traffic and Electricity datasets—both of which feature strong, predictable seasonal patterns—yield similar "fingerprints" across the network layers, whereas unrelated datasets like Weather look structurally different.
Future Directions
While these models operate in an extrinsic dimension of $D=512$, this research proves the data actually lives on a manifold with an ID between just 1.2 and 8.1. However, the relatively flat "slope" of the relationship between curvature and error across different datasets suggests that current architectures might not be expressive enough to fully capture the complexity of datasets with hundreds of features.
Moving forward, these geometric tools open exciting new avenues for TSF research. Future work could involve designing novel loss functions that explicitly penalize high manifold curvature to improve generalization, or using these profiles as a diagnostic tool to compare emerging architectures (like PatchTST) without relying solely on traditional benchmark scores.
Summary updated