Introduction
Deep learning models, particularly Fully Convolutional Networks (FCNs), have established themselves as the state-of-the-art for Time Series Classification (TSC). However, a persistent challenge in this domain is the disparity between model capacity and data availability. Many TSC datasets are relatively small, leading to over-parameterization where networks possess far more degrees of freedom than necessary. This paper identifies a specific consequence of this over-parameterization: filter redundancy, and proposes a novel regularization technique to mitigate it.
The Problem: Correlation Drift
A common practice in deep learning is to use Orthogonal Initialization to ensure that convolutional filters start as unique, uncorrelated feature detectors. However, the authors observe a phenomenon they term "Correlation Drift." As training progresses, filters that were initially unique tend to become highly correlated, learning to identify the same patterns. This redundancy effectively shrinks the network's capacity and increases the risk of overfitting.
As shown in the figure above, standard Batch Normalization (BN) allows filter correlation to spike significantly across all layers. When multiple filters converge on the same features, the model fails to capture the diverse range of temporal patterns necessary for robust classification.
Proposed Method: Decorrelation Loss
To combat this drift, the authors introduce a Decorrelation Loss ($L_{Decorr}$) that acts as a mathematical tether, forcing filters to remain distinct. The architecture remains a standard 3-layer FCN, but the training objective is modified:
- Pairwise Similarity: For each layer, the cosine similarity between every possible pair of filters is calculated.
- Penalty Calculation: These similarity scores are squared to penalize both positive and negative correlations and then summed across all layers.
- Integrated Objective: This term is weighted by a hyperparameter ($\lambda$) and added to the standard Cross-Entropy loss: $L = L_{CE} + \lambda L_{Decorr}$.
By minimizing this joint loss, the network is incentivized to maintain a diverse set of feature extractors throughout the entire training process.
Experimental Evaluation
The researchers evaluated the approach using 44 univariate time series datasets from the UCR TSC benchmark repository. To understand the impact of data scarcity, the datasets were categorized into three groups based on the ratio of training instances ($I$) to labels:
- Small: $I \leq 11$
- Medium: $11 < I \leq 33$
- Large: $I > 33$
Key Results and Insights
The experimental results demonstrate that the BN+Decorr approach consistently outperforms standard Batch Normalization and the common BN+Dropout configuration.
Performance Gains in Data-Scarce Scenarios
The most dramatic improvements were observed in the "Small" dataset category, where the average error rate dropped from 0.2961 (BN) to 0.2509 (BN+Decorr). This suggests that decorrelation is a particularly powerful tool for low-resource time series tasks. In stress tests using only 30% of available training data, the regularizer provided an 8.61% performance gain over the baseline.
The Failure of Dropout
Interestingly, the study found that Dropout significantly worsened performance in this FCN context. While Dropout is a staple for regularizing dense layers, applying it to convolutional filters in these specific TSC models led to much higher error rates compared to simple Batch Normalization. This highlights the importance of choosing regularization techniques tailored to the specific architecture and data type.
Conclusion
By penalizing filter correlation, the proposed method prevents the "Correlation Drift" that plagues over-parameterized FCNs. This simple yet effective regularization technique ensures that networks maintain their effective capacity, leading to significant accuracy improvements, especially on small-to-medium datasets where overfitting is most prevalent. Future work will look to extend these findings to multivariate time series and explore the potential for more interpretable, "disentangled" feature representations.
Summary updated