How Much of a VAE Feature's Value Is the VAE? A Controlled Audit on Sensor and ECG Benchmarks

Many applied papers in manufacturing, cardiology, fault diagnosis and forecasting use a variational autoencoder (VAE) as an automatic feature extractor. A VAE is a neural network that squeezes each input through a small, noisy, probabilistic bottleneck and is trained to reconstruct the input from it. These papers then report that the compressed features beat hand-engineered ones. Each paper, however, picks its own data split, its own downstream classifier, its own random seed, and its own way of reading features out of the encoder. The benchmarking literature has shown repeatedly that choices like these can produce the entire claimed gain.

We propose a factorial re-evaluation of VAE feature extraction on time-series benchmarks that already have leak-free splits. The question is narrow: once split, downstream classifier capacity, seed and tuning budget are held fixed, how much downstream performance is attributable to the VAE's defining ingredient, the KL penalty that turns a plain autoencoder into a variational one? A second question is whether practitioners could pick a good VAE checkpoint without labels at all.

Core Claim

The claim we will test, and are prepared to see refuted, has two parts. First, on patient- or unit-held-out splits with a fixed linear probe, the advantage of VAE features over a deterministic autoencoder of matched size and over PCA is small compared with the spread caused by seeds and probe choice. Second, any real VAE advantage concentrates in the low-label regime and in features that use the posterior variance, which almost every applied paper throws away.

The deliverable is a variance decomposition of downstream utility. It is modeled on the one Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations produced for disentanglement metrics. That work found the objective explained about 37% of score variance, rising to 59% with hyperparameters. Ours would be computed for downstream prediction on real signals rather than for disentanglement scores on synthetic images.

Why This Gap Is Real

The motivating evidence comes from two sides.

On the applied side, several papers report the gains in question. Dynamic prediction of malignant ventricular arrhythmias feeds 16 VAE latents from ECGs into a random survival forest and reports a time-varying AUROC of 0.738 ± 0.07 versus 0.639 ± 0.03 for static clinical variables. It uses an 80:20 split over ECGs and does no external testing. Rolling Bearing Fault Diagnosis Based on Deep Learning and Autoencoder Information Fusion pairs VAE latents with a random forest and reports 98.19% accuracy, and 94.33% with 4.92% of training labels. Cross-Domain Offshore Wind Power Forecasting clusters VAE weather latents to guide Gaussian-process transfer. Each paper pairs the features with a different nonlinear downstream model. None isolates the KL term or reports seed variance.

On the methodological side, other work warns that such gains can be protocol artifacts. Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline showed that controlling augmentation, loss and checkpoint selection lifts PointNet++ from 90.0% to 93.3% and lets a simple projection baseline win. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL showed that record-level random splits leak patients and inflate AUC. SentEval fixed probe classifiers and tuning grids precisely so that embeddings, not evaluation scripts, are compared.

The closest existing works each leave this question open:

  • Locatello et al. ran the right kind of controlled study, but on dSprites-style images with known factors. Their downstream results showed no consistent benefit from higher disentanglement, which constrains our hypothesis. They also identified the mean-versus-sample disconnect: training regularizes sampled latents, yet everyone extracts the posterior mean, whose dimensions stay correlated. We carry both findings to real sensor and clinical signals.
  • EEG-FM-Bench standardizes frozen-probe and fine-tuning evaluation and finds that reconstruction pretraining gradients conflict with downstream tasks. It covers masked-transformer foundation models, not VAEs, and does not ablate the probabilistic bottleneck.
  • ConTSG-Bench benchmarks VAE-family generators, but for conditional generation. It does not test frozen encoders as features.

What none of these does is hold the encoder architecture fixed, vary only the KL weight and the feature readout, and measure the downstream consequence on leak-free time-series splits.

Research Program

Encoder families. We build one shared one-dimensional convolutional residual encoder and train it in several ways:

  • as a deterministic autoencoder (KL weight zero, with weight decay matched to the VAE);
  • as a VAE with KL weight $\beta \in \{0.1, 1, 4\}$;
  • with a bottleneck of 16 or 64 dimensions. Sixteen matches the Kolk et al. setting.

Non-learned comparisons. Alongside the learned encoders we compute PCA at the same dimensions and the wavelet-coefficient features used as the hand-crafted baseline in the PTB-XL benchmark paper.

Feature readouts. For every VAE we read features out in three ways:

  • the posterior mean $\mu(x)$, which is standard practice;
  • a single posterior sample;
  • the concatenation of $\mu(x)$ with the log posterior standard deviation.

The third readout tests whether the encoder's own uncertainty carries signal that current pipelines discard.

Diagnostics on the latents. We record three label-free statistics for each trained encoder: the off-diagonal correlation of the mean vectors, the number of "active" latent dimensions (those whose mean varies across inputs), and held-out ELBO.

Downstream probes. Every feature set is evaluated with three probes, each with a fixed hyperparameter grid shared by all feature families:

  • L2-regularized logistic regression;
  • a one-hidden-layer MLP;
  • a random forest.

Splits, seeds and label budgets. Each configuration is trained with 10 seeds. Each is evaluated under the benchmark's leak-free split and, as a deliberate contrast, a record-level random split. Each is also evaluated at 1%, 5% and 100% of downstream labels.

Analysis. We fit a mixed-effects variance decomposition over these factors: encoder family, KL weight, readout, probe, split, label fraction and seed. We also test whether the label-free statistics rank checkpoints in the same order as downstream AUC. This directly addresses the open complaint that applied VAE pipelines have no principled way to select a checkpoint without labels.

Baselines

Naive reference points. The floor is a probe trained on the downsampled raw signal. A second floor is the wavelet-coefficient pipeline from the PTB-XL benchmark paper, which that paper reports "lagged significantly behind" deep models; no reported number found on Lacuna for its AUC. Both use the same probes and grids as every learned feature set, so any gain over them is attributable to the features.

PCA. PCA at matched dimension is the strongest cheap competitor. VFMF reports it degrading DINOv2 features badly (ImageNet FID 18.49 with PCA versus 11.76 with a VAE, VFMF). That evidence concerns generation, not prediction, so it does not settle the question here.

Strongest existing method. This is the supervised end-to-end convolutional ensemble on PTB-XL, which reaches a macro AUC of approximately 0.939 on diagnostic statements under the patient-stratified split (Deep Learning for ECG Analysis). We retrain the same architectures on our label fractions with the same epoch budget used to train our encoders plus probes. This bounds what frozen features can achieve.

For bearings, the reference is VAERF's 98.19%. Its split protocol is not described in the file, so we re-implement it under our held-out-load split rather than quoting it as comparable. For the arrhythmia cohort the data are not public, so the 0.738 AUROC cannot be re-run and serves only as motivation.

Key ablation. The ablation that isolates the proposal's central ingredient is the deterministic autoencoder: same encoder, same bottleneck, same compute, with the KL weight set to zero.

What counts as winning. A VAE configuration "beats" a baseline only if the gap exceeds the across-seed standard deviation under the linear probe on the leak-free split. Every family receives the same hyperparameter-trial budget.

Why not just…?

Why not just use PCA and skip the neural network? This may well be the right answer for linear probes at full labels. The audit is designed so that this outcome is reportable rather than embarrassing. The evidence favoring VAEs over PCA in the file comes either from generation quality (VFMF) or from a single bearing study with nonlinear downstream classifiers. Neither separates feature quality from probe capacity, which is exactly the confounder our probe factor measures.

Why not just use a deterministic autoencoder, which avoids posterior collapse? This is the real threat to the VAE-specific claim. Locatello et al. found no consistent downstream benefit from stronger probabilistic regularization. Our $\beta = 0$ arm with matched weight decay settles the question directly. If the autoencoder ties the VAE across label fractions, we will conclude that the KL term contributes nothing measurable to feature utility on these tasks. The one remaining VAE-specific advantage would then be the posterior-variance readout, if it helps at all.

Why not just train end-to-end? EEG-FM-Bench reports large drops for frozen backbones relative to fine-tuning. We expect end-to-end training to win at full labels on PTB-XL. The interesting question is the 1% and 5% regimes, where the bearing study claims unsupervised features rescue performance. Our label-fraction sweep measures where, if anywhere, the curves cross.

Why not just reuse disentanglement_lib or EEG-FM-Bench? The first is built around synthetic images and disentanglement metrics. The second evaluates masked-transformer models and never varies a probabilistic bottleneck. Neither answers whether the KL term matters for real signal features.

Proposed Experiments

Datasets.

  • PTB-XL with its patient-stratified folds is the primary testbed, reporting macro AUC on diagnostic statements.
  • ICBEB2018 tests transfer of PTB-XL-trained encoders, as in the PTB-XL benchmark paper.
  • The CWRU bearing data are re-split so that test files come from operating loads unseen in training. This removes overlap between windows cut from the same recording.
  • Three EEG-FM-Bench datasets (motor imagery BCIC-2a, sleep staging, and seizure detection) use that benchmark's preprocessing and report balanced accuracy.

Decisive checks. Four results would decide the claim:

  1. The share of downstream-metric variance explained by KL weight, compared with the shares explained by seed and probe. Our claim fails if KL weight dominates.
  2. The inflation from record-level versus held-out splits for VAE features specifically. If VAEs inflate more than PCA, leakage flatters them in particular.
  3. Whether the mean-plus-log-variance readout beats the mean alone beyond seed noise.
  4. The Spearman correlation between each label-free selection statistic and downstream AUC across checkpoints, compared against random checkpoint choice.

Taken together, these show which parts of the applied VAE literature's gains survive a controlled protocol. They also give practitioners a tested rule, or an honest absence of one, for choosing an encoder without labels.