Choosing the Noise: An F-Ratio for Test-Time Stochastic Resonance

Stochastic resonance is the effect where adding the right amount of random noise to a system with a threshold makes a weak signal easier to detect. Too little noise and the signal never crosses the threshold. Too much and the output is swamped. Machine learning has recently used this effect at test time. Sub-token ViT Embedding via Stochastic Resonance Transformers shifts an image by fractions of a patch, runs a frozen vision transformer (an image model that cuts pictures into fixed 16×16 tiles, or "tokens") on each shifted copy, undoes the shifts on the feature maps, and averages them. This recovers detail finer than one tile. Improving Pursuit Algorithms Using Stochastic Resonance does the same for sparse signal recovery: it adds noise to the input of a greedy solver, runs it $K$ times, and averages the results. Both methods need two numbers that nobody knows how to set without labels: how much noise to add, and how many noisy passes $K$ to pay for.

We propose to treat those $K$ passes as what they already are, a small designed experiment, and to analyze them with analysis of variance (ANOVA, the classical test of whether differences between group averages exceed the scatter within groups). For each token, the noise levels are the groups. The between-group term measures how far noise moves the averaged feature away from the noiseless feature. The within-group term measures how much the individual noisy passes disagree. Their ratio, the Fisher $F$-statistic, is a label-free version of the classical "output signal-to-noise ratio" of stochastic resonance. We claim that it traces the inverted-U resonance curve for each token and each image. If so, it can pick the noise level without labels and can decide which tokens deserve more passes, so the fixed $K$-fold cost of test-time stochastic resonance becomes a budget spent only where noise helps.

Core Claim

Let $f(p;\delta)$ be the frozen model's feature at location $p$ after perturbation $\delta$, drawn with scale $\sigma$ (the sub-token shift magnitude, or the input noise level for pursuit). Let $f(p;0)$ be the unperturbed feature. With $K$ passes, the quantity that governs whether noise did anything detectable is

$$ F_p(\sigma) \;=\; \frac{K\,\lVert \bar f_K(p;\sigma) - f(p;0)\rVert^2 / d}{\widehat{\mathrm{Var}}_\delta\, f(p;\delta)}, $$

where $\bar f_K$ is the average over the $K$ passes, $d$ is the feature dimension, and the denominator is the average per-dimension spread across passes. This is the one-way ANOVA ratio with the noiseless feature as the reference group.

The inverted U is predicted by a Taylor-expansion argument. For small symmetric perturbations, first-order effects cancel in the average, so the numerator grows like $\sigma^4$ while the spread grows like $\sigma^2$, and $F$ rises. For large perturbations, the mean shift saturates because features are bounded, while the spread keeps growing through content change and boundary artifacts, so $F$ falls.

The empirical claim has two parts. First, the $\sigma$ that maximizes a pooled $F$ lands close to the $\sigma$ that maximizes labeled downstream accuracy. Second, tokens whose $F$ is not significant after a few passes can stop early with negligible loss.

The ratio is essential, not decorative. Raw pass-to-pass variance rises monotonically with $\sigma$ and so cannot pick a noise level. Raw mean shift also rises until saturation. Only their ratio has an interior optimum.

Relation to Prior Work

Two papers supply the method we build on and motivate the gap. Sub-token ViT Embedding via Stochastic Resonance Transformers shows that averaging intermediate embeddings over sub-token shifts improves frozen models without training. Its shift magnitude and number of passes are fixed by hand, and it notes both the multi-pass latency and boundary artifacts. Improving Pursuit Algorithms Using Stochastic Resonance shows non-monotone error against noise level, and it points to an external risk estimator (Stein's unbiased risk estimate, SURE) as the route to choosing noise. We keep both perturbation schemes unchanged and replace their hand-set parameters with a statistic computed from the passes themselves.

Activity Driven Adaptive Stochastic Resonance is the closest conceptual ancestor. A model neuron tunes its own noise variance toward the optimum using only local firing statistics. We carry that "tune from local statistics" idea from a single neuron to each token of a deep network, and we use a significance test rather than a gradient rule, so the stopping decision has an explicit error rate.

Functional ANOVA (fANOVA, which splits a performance surface into contributions from each hyperparameter and their interactions) is the other obvious neighbor. An Efficient Approach for Assessing Hyperparameter Importance and Hyperparameter Importance Across Datasets apply it offline, to labeled performance across many configurations. We use fANOVA only in a secondary analysis. Our core use of ANOVA is online, per input, and needs no labels, which is a different job.

One tension in the stochastic-resonance literature constrains our design. Deliberate noise inflates within-group variance, which weakens an ordinary $F$-test. We turn this into the mechanism itself: the denominator is exactly what the averaging must beat. Wei Xu shows that heavy-tailed jump noise can drive resonance, which warns that the Gaussian $F$ reference distribution may be miscalibrated. We therefore calibrate thresholds with permutation over pass assignments rather than trusting $F$-tables.

Research Program

We will build a drop-in wrapper, starting from the public SRT code (github.com/donglao/srt), that proceeds in three steps. First, it runs a small pilot of $K_0$ passes at each of a handful of noise levels on the image. Second, it computes the token-wise $F$-ratio and pools it by median over tokens, then selects $\sigma^\star$ as the level with the largest pooled $F$. Third, it runs a sequential test at $\sigma^\star$. Tokens that fail significance at a false-discovery-controlled threshold keep their unperturbed feature. The remaining passes are spent on the significant tokens, which are processed as cropped windows with context padding, until the averaged feature's standard error falls below a set tolerance. This follows from the $\mathrm{Var}/K$ scaling of an average of independent passes.

We will run the same wrapper on noise-injected Subspace Pursuit, with image patches as the units. This gives a second testbed where a principled label-free competitor, SURE, exists.

In a secondary analysis we will apply fANOVA over shift magnitude, $K$, the aggregation operator (mean versus median), and backbone layer, using labeled downstream scores. The question is whether noise level acts mainly through interactions with layer, as Gap 1 of the research file suspects. We will then check whether per-image $F$ captures that interaction.

Baselines

The do-nothing reference is the frozen backbone in a single pass: plain ViT features with a linear head on NYU-Depth V2 and DAVIS-2017, and single-run deterministic Subspace Pursuit for denoising. For the latter, Improving Pursuit Algorithms Using Stochastic Resonance reports 26.88 dB PSNR on the Chinese Passport dataset.

The strongest existing method is fixed-budget stochastic resonance with hand-tuned noise. For the transformer, this is SRT run with its authors' settings. Sub-token ViT Embedding via Stochastic Resonance Transformers reports up to a 14.9% improvement in depth root-mean-square error (RMSE) on NYU-Depth V2 with a linear head, and a 4.1% relative gain on DAVIS-2017 for ViT-S/16. For pursuit, it is noise-averaged Subspace Pursuit at $K=25$ or $50$, which reaches 28.76 dB. We will rerun both on identical splits, backbones, and heads. The reference noise level will be tuned on labeled validation data with the same grid we give our pilot.

For pursuit noise selection, the strongest label-free competitor is SURE-based choice of $\sigma$. The file reports no result on Lacuna for SURE-tuned stochastic resonance pursuit, so we will implement it ourselves.

Two ablations isolate the key ingredient. The first replaces the $F$-ratio with raw pass-to-pass variance, used for token allocation. For noise selection, raw variance has no interior optimum, which is itself a check. The second allocates the same number of extra passes to randomly chosen tokens.

We count a win if either of two things holds. One is matching fixed-$K$ SRT accuracy, within seed-level confidence intervals, at no more than half its forward-pass FLOPs. The other is beating label-tuned fixed $\sigma$ when the tuning set comes from a different dataset than the test set. All comparisons get the same total pass budget.

Why not just…?

Why not just tune $\sigma$ once on a labeled validation set and fix $K$? This is the real incumbent, and on in-distribution data it may be hard to beat on accuracy. Our advantage has to show up in two places. One is compute, since flat regions stop early. The other is transfer: we will tune the fixed $\sigma$ on NYU-Depth V2 and test on ADE20K and DAVIS-2017, where the per-image $F$ can adapt and the fixed setting cannot. If the fixed setting transfers as well and the savings are small, we will report that the adaptivity is unnecessary.

Why not just threshold the raw variance across passes? For deciding which tokens to refine, this may work nearly as well, and the ablation will say so directly. It cannot choose the noise level, because variance grows monotonically with $\sigma$. The within-pass normalization matters most where boundary artifacts inflate variance without adding signal, and we will report the ablation separately for image-border tokens.

Why not just distill the ensemble into one network? Distillation is complementary, not a substitute. Sub-token ViT Embedding via Stochastic Resonance Transformers reports that distilling keeps single-pass speed but retains only a 1.3% boost on DAVIS-2017, compared with 4.1% for the full ensemble. It also requires training. Our wrapper stays training-free. We will additionally test whether distilling our adaptively chosen targets closes more of that gap.

Why not just use SURE? For pursuit denoising under a known Gaussian noise model, SURE may well win, and we treat that setting as a calibration check rather than the target. SURE does not exist for frozen-feature tasks such as depth estimation or segmentation. The $F$-ratio does, and extending to those tasks is the point.

Proposed Experiments

The decisive check is whether label-free $\sigma^\star$ matches label-optimal $\sigma$. We will sweep shift magnitude densely on NYU-Depth V2 (Indoor Segmentation and Support Inference from RGBD Images), DAVIS-2017 (The 2017 DAVIS Challenge on Video Object Segmentation), and ADE20K. The backbones are CLIP, DINO, and MAE ViTs, the same frozen pretrained models the SRT paper evaluated. We will measure the per-image rank correlation between pooled $F$ and labeled error across $\sigma$. We will also measure the accuracy lost by choosing $\sigma^\star$ instead of the oracle $\sigma$. If the pooled $F$ peak does not track labeled accuracy, the central claim fails. That result would still be informative, because it would show that detectable noise-induced change and useful change diverge in deep features.

The second check is compute versus accuracy. We will plot RMSE, $\mathcal{J}\&\mathcal{F}$, and mean intersection over union (mIoU) against total forward-pass FLOPs for three allocators at matched budgets: adaptive $F$-based allocation, fixed $K$, random-token allocation, and variance-threshold allocation.

The third check, on the Chinese Passport patches, compares $F$-selected noise against SURE and against the oracle in PSNR, starting from the reported 26.88 and 28.76 dB anchors.

Finally, every reported gain will come from a mixed-effects ANOVA with perturbation seed and image as random effects. The permutation-calibrated false-discovery rate of the token test will also be checked under heavy-tailed shift distributions, following the concern raised by Wei Xu. This way, claimed resonance gains are separated from seed-to-seed scatter, a separation the existing point comparisons do not provide.