Timescale-Diverse Slow Feature Ensembles

Slow feature analysis (SFA) learns representations by asking which functions of a signal change least from one moment to the next. The bet is that the things we care about, such as a heart's underlying condition or an object's identity, drift slowly while noise flickers quickly. Ensembling, meaning averaging the predictions of several models, only helps when the members make different mistakes. These two ideas pull against each other. Classical SFA has a unique answer: in the linear case it is the solution of a single eigenvalue problem. So several networks trained for slowness on the same data should converge to nearly the same features. If that happens, an ensemble of them is close to one model run several times. This note proposes to measure that collapse directly, and to replace the usual source of ensemble diversity, different random seeds, with a source that slowness itself provides: the timescale over which "slow" is defined.

The concrete proposal is to train a small committee of self-supervised ECG encoders on PTB-XL that are identical except for their slowness horizon. The horizon is how far apart in time two windows of a recording may be while the encoder is still asked to map them to the same features. We then test whether this committee disagrees more usefully than a seed ensemble, and whether it classifies better at matched compute. A colleague's two-sentence version: slowness-trained models all find the same slow subspace, so seed ensembles of them are nearly redundant. Different timescales define genuinely different slowness problems, so a committee indexed by timescale should make errors that cancel when averaged.

Core Claim

Write the linear SFA problem at horizon $\tau$ as

$$A_\tau w = \lambda B w, \qquad A_\tau = \mathbb{E}\big[(x(t+\tau)-x(t))(x(t+\tau)-x(t))^\top\big],\quad B = \mathbb{E}[x x^\top].$$

Here $x(t)$ is the input at time $t$, $w$ is a feature direction, and $\lambda$ measures how fast that feature changes. This is the generalized eigenvalue form described by Bray et al., with the one-step difference replaced by a $\tau$-step difference. For fixed $\tau$ the slowest eigenvectors are unique up to sign and rotation within tied eigenvalues. Changing the seed therefore cannot make linear SFA members diverse. Changing $\tau$ changes $A_\tau$ and so changes the answer.

We claim this carries over to deep encoders trained with slowness-style contrastive losses. Specifically, seed ensembles of such encoders show lower disagreement-to-error ratios than seed ensembles of supervised networks. Horizon-indexed ensembles restore that ratio and turn it into accuracy gains. Representation similarity, measured with centered kernel alignment (CKA, a similarity score between two networks' internal features), should be the mechanism: high across seeds and lower across horizons.

This is different from the closest work. Learning ECG Representations via Poly-Window Contrastive Learning invokes slowness within one network at one fixed window scale, with a 0.64-second optimum, and never ensembles. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL ensembles supervised CNNs with no slowness objective. Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors ensembles uniform next-step RNNs and never measures member diversity.

Two prior results motivate the prediction that slowness ensembles are fragile. When are ensembles really effective? shows ensemble gains track the disagreement-error ratio (DER) and vanish below 1. Maximizing Ensemble Diversity in Deep Reinforcement Learning shows independently seeded networks synchronizing to CKA of 0.96–0.97. Neither looked at slowness objectives, where the theory predicts the synchronization should be strongest.

Research Program

The work has three stages. The first stage is a controlled sanity check on the synthetic testbed of A Biologically Plausible Neural Network for Slow Feature Analysis, which hides a multi-sine driving force inside a noisy logistic map. There we solve linear and expanded-quadratic SFA exactly at several horizons and with several bootstrap resamples. We then measure how much the recovered features vary within a horizon versus across horizons. This stage checks the eigenproblem argument where it can be computed exactly. It also tells us whether horizon diversity yields features that complement each other for recovering the hidden signal, or just noisier copies of it.

The second stage is the main study on PTB-XL. We train encoders with the Poly-Window objective of Yuan et al. One change is made: positive pairs are windows whose start times differ by at most $\tau$. We sweep $\tau$ over roughly 0.3 to 8 seconds, spanning sub-beat morphology, single cardiac cycles, and multi-beat rhythm. Crop length is held fixed. This is deliberate, so that horizon is not confused with the amount of signal each window sees.

For each horizon we train several seeds. We then compute pairwise DER, CKA, and prediction correlation on the patient-stratified test split. Ensembles are formed by averaging probe outputs from:

  • members sharing one horizon but differing in seed;
  • members spread across horizons;
  • a greedy subset chosen on validation data to maximize DER.

The third stage tests whether horizon diversity is complementary to supervised diversity. We mix slowness members with supervised CNN members and ask whether the mixed committee beats an all-supervised committee of the same size. This is the regime where a ceiling of strong supervised ensembles would actually be raised.

Baselines

The do-nothing reference is a single Poly-Window encoder at its reported best configuration, evaluated with the same linear probe. Learning ECG Representations via Poly-Window Contrastive Learning reports 0.891 AUROC and 0.680 F1 on PTB-XL superclass classification with eight windows. We will reproduce this. The proposal must beat our reproduced number, not the quoted one.

The direct competitor for the key ingredient is a seed ensemble: $K$ Poly-Window encoders at the best single horizon, differing only in initialization and data order. They use the same total pre-training budget and the same probe tuning budget as the horizon ensemble. Beating it means a higher macro AUROC at equal $K$, with a DER gap in the predicted direction. If the horizon ensemble wins on AUROC but not on DER, the mechanism claim fails even though the method works.

The strongest existing method is the supervised CNN ensemble of Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL. It reports a macro AUC of about 0.939 on diagnostic statement prediction. That is a different label set from the superclass task, so the two published numbers are not comparable. We will retrain the supervised ensemble on both tasks with the same $K$ and training compute. Beating it in the full-label regime is not expected. The relevant wins are two: a mixed committee exceeding the all-supervised committee, and horizon ensembles closing the gap with 1% and 10% of labels.

Two ablations isolate the ingredient. The first is a crop-length ensemble: horizon fixed, window length varied across members. This tests whether any multi-scale variation produces the effect. The second is an ensemble trained with the Adaptive Diversity Promoting regularizer of Improving Adversarial Robustness via Promoting Ensemble Diversity. That method forces members' non-predicted class probabilities to be mutually orthogonal, and it is the most direct generic way to buy diversity explicitly. We apply it during probe training over seed-ensemble members.

Why not just…?

Why not just use a seed ensemble of the best slowness encoder? Because the eigenproblem argument predicts it is close to redundant. The seed-collapse evidence of Maximizing Ensemble Diversity in Deep Reinforcement Learning points the same way, though for a different objective. This is an empirical question, not a settled one. The stage-two seed-versus-horizon CKA and DER comparison settles it. If seed ensembles already reach DER above 1 with high gains, the motivating premise is wrong and we will report that.

Why not just train one encoder with several horizons and average its heads? This is the most serious threat. A single multi-horizon model with matched parameters is cheaper at inference and might capture the same multi-scale information. We train exactly this model, with one shared trunk, one head per horizon, and the summed loss, at the same total parameter count and compute as the $K$-member committee. If it matches the committee, the honest conclusion is that multi-timescale features matter and separate models do not. The proposal would then become a study of multi-horizon slowness rather than of ensembling.

Why not just add a generic diversity penalty such as ADP? Explicit orthogonality strongly helped adversarial robustness in Improving Adversarial Robustness via Promoting Ensemble Diversity, lifting white-box PGD accuracy on MNIST from 6.3% to 41.0%. That penalty, however, acts on output probabilities and can buy disagreement without useful complementary features. The ADP baseline above tests this directly. Horizon diversity should win on clean AUROC per unit of DER. ADP may win on adversarial metrics, which we report separately rather than hide.

Why not just train the supervised CNN ensemble and stop? In the full-label regime it likely wins outright. The proposal's value lies in the low-label and mixed-committee settings, and the third stage measures both.

Proposed Experiments and Decisive Checks

All PTB-XL experiments use the standard patient-stratified folds at 100 Hz. Strodthoff et al. report no significant diagnostic gain from moving to 500 Hz. We report macro AUROC on superclass and all-statement tasks, along with expected calibration error, pairwise DER, ensemble improvement rate, and CKA between members, using $K \in \{2,4,8\}$ and probes trained on 1%, 10%, and 100% of labels.

For robustness, we add synthetic baseline wander and high-frequency noise at test time. This yields a corruption benchmark of our own design; no reported result was found on Lacuna for corrupted PTB-XL. These corruptions are interesting because they attack slow and fast horizons differently.

Four outcomes are decisive:

  • Within-horizon CKA should exceed across-horizon CKA, and seed-ensemble DER should be lower for slowness encoders than for supervised CNNs. Failing either breaks the mechanism.
  • The horizon ensemble should beat the seed ensemble and the crop-length ensemble at equal $K$. Failing this breaks the method.
  • The horizon ensemble should beat the matched single multi-horizon model. Failing this collapses the claim to multi-timescale features alone.
  • Adding horizon members to a supervised committee should beat adding further supervised members. This determines whether the idea matters in practice.

The synthetic stage, where SFA is solved exactly, makes a falsifiable prediction before any deep training. If exact SFA shows no horizon-dependent variation in the recovered features, the PTB-XL study is not worth running in this form.