Abstract
Standard time-series evaluation often conflates sequence length with statistical information content, leading to biased conclusions about model performance. This paper introduces a fair-comparison methodology based on effective sample size ($N_{eff}$) and provides the first architecture-aware generalization bounds for Temporal Convolutional Networks (TCNs). By explicitly accounting for temporal mixing and network structure, we demonstrate that temporal dependence can actually enhance generalization when information content is held constant. Our theoretical bounds show that TCN complexity scales with the square root of depth and the logarithm of kernel size, a result supported by empirical findings on synthetic and real-world datasets.
Introduction
In the study of independent and identically distributed (i.i.d.) data, the sample size $N$ is a direct proxy for the information budget. However, in time-series analysis, temporal dependencies mean that $N$ observations may contain significantly less information than $N$ independent samples. This leads to two critical issues: evaluation bias, where models are compared on sequences of equal length but unequal information content, and a lack of theoretical guarantees that account for specific temporal architectures like TCNs.
Methodology
The Fair Comparison Protocol
To address evaluation bias, we propose matching the Effective Sample Size ($N_{eff}$) rather than the raw sequence length $N$. For a given stochastic process, $N_{eff}$ quantifies the equivalent number of independent observations. In our experiments with AR(1) processes, we adjust the sequence length for high-dependency regimes (high $\rho$) to ensure that every model is trained on an equivalent "information budget."
Theoretical Framework: Blocking and Anchors
We derive generalization guarantees for TCNs by employing a blocking and coupling technique. The dependent sequence is partitioned into blocks, and the first element of each block is designated as an "anchor." By ensuring sufficient spacing between these anchors—determined by the mixing rate of the process—they can be treated as approximately independent for the purpose of bounding Rademacher complexity.
Our framework utilizes $\ell_{2,1}$ filter-group norms to bound the capacity of the TCN. This approach reveals that complexity scales sublinearly with depth ($\sqrt{D}$) and logarithmically with kernel size ($\log p$), preventing the exponential complexity explosion often feared in deep temporal models.
Empirical Results
The "Information vs. Structure" Reversal
A key finding of this work is the reversal of standard conclusions when using fair comparison. Under a fixed sequence length $N$, stronger temporal dependence appears to hinder learning as the generalization gap grows. However, when holding $N_{eff}$ constant, stronger dependence actually reduces the generalization gap by approximately 76% when moving from $\rho=0.2$ to $\rho=0.8$. This suggests that TCNs possess an inductive bias that effectively exploits temporal regularities.
Scaling and Convergence
Our theoretical bound is established as $O(\sqrt{D \log p / B})$, where $B$ is the number of independent anchors. Empirically, on AR(1) data, generalization gaps decayed at rates between $N_{eff}^{-0.89}$ and $N_{eff}^{-1.21}$. These rates are significantly faster than the standard $N^{-1/2}$ theoretical worst-case, indicating that modern temporal architectures are highly efficient at capturing structured dependencies.
Conclusion
This research highlights that sequence length is a poor proxy for data quantity in temporal contexts. By shifting the focus to effective sample size and architecture-aware bounds, we provide a more accurate framework for evaluating and understanding temporal networks. Future work will focus on developing robust $N_{eff}$ estimators for non-stationary real-world data and extending these theoretical insights to attention-based architectures like Transformers.
Summary updated