Matching Memory Horizons: Training-Free Architecture Scoring for Neural Time Series

Brain-signal decoders work best when their architecture fits the person and the task. One published result shows this sharply. On a four-class motor-imagery EEG benchmark (EEG recorded while people imagine moving a hand, foot or tongue), giving each subject their own searched architecture raised accuracy from about 77–79% to 88.52%. The method that achieved this trains a large "super-network" for every subject, and that is too expensive to repeat for each new user of a wearable brain-computer interface. Cheap alternatives exist in the form of zero-cost proxies: formulas that score a candidate network from one or two forward and backward passes at random initialization, with no training. These formulas were designed for image convolutional networks, and they are known to break on time series.

We propose a training-free score built on a simple question: how far back in time does a randomly initialized network listen, and does that match how far back the signal carries useful information? We measure the first part by asking how strongly the network's output at the final time step responds to the input $k$ steps earlier, at initialization. We estimate the second part from the subject's own calibration data using cheap linear decoders. Architectures whose sensitivity-over-lag profile matches the data's information-over-lag profile, and whose gradients neither vanish nor explode over that span, should rank highest. If the score works, per-subject architecture search drops from GPU-days to minutes. It would also be the first proxy that can compare convolutional, recurrent, attention and ODE blocks on one scale.

Core Claim

A network's temporal sensitivity profile at initialization predicts how it will rank after training on neural time series, and it predicts better than existing proxies. The profile has to be compared against a task-specific dependency profile measured from data. The new element is this matching to the data's own horizon. Existing proxies score a network in isolation, and existing dynamical-isometry results only ask whether gradients survive. Neither asks whether the network's memory horizon is the right one for the task.

Concretely, for an architecture $A$ at random initialization and input windows $x_{1:T}$ from the subject's calibration set, define the sensitivity profile

$$s_A(k) = \mathbb{E}_{x}\,\big\|\partial y_T / \partial x_{T-k}\big\|_F, \qquad k = 0,\dots,T-1,$$

normalized to sum to one. Here $y_T$ is the output at the last step and the expectation runs over a few dozen windows. The data dependency profile $d(k)$, also normalized, is the marginal gain in held-out accuracy of a ridge or logistic decoder when its input window grows from $k$ to $k+1$ steps. Fitting these decoders takes seconds. The score is

$$S(A) = -\,\mathrm{JS}\big(s_A \,\|\, d\big) \;-\; \lambda\,\big|\log \bar\sigma_A\big|,$$

where $\mathrm{JS}$ is the Jensen–Shannon divergence between the two profiles. $\bar\sigma_A$ is the geometric mean singular value of the end-to-end input-output Jacobian over the window, a dynamical-isometry term that penalizes vanishing or exploding signal. $\lambda$ is a single weight tuned on held-out subjects. The first term asks whether the network is listening at the right lags. The second asks whether it can learn at all.

Position Relative to Prior Work

Three papers motivate and constrain the gap. NAS-Bench-360 benchmarks architecture search methods and vision proxies on non-vision tasks such as NinaPro EMG and ECG. It finds that CIFAR-100 rankings carry almost no information about these tasks and that vision proxies are volatile there. This is the evidence that a temporal proxy is needed. LPZero evolves symbolic proxies for Transformers and shows hand-built proxies collapse outside convolutional networks: SNIP reaches Spearman 0.173 on FlexiBERT versus 0.748 for its evolved proxy. EZNAS evolves proxies over ReLU-Conv-BatchNorm statistics and reaches Kendall 0.65 on NAS-Bench-201. Both of these evolved proxies are tied to one family of building block and have no notion of lag. Our score is defined for any block that maps a sequence to an output.

Two papers supply the method. Gilboa et al. show with mean-field theory that the temporal Jacobian spectrum and the forget-gate bias decide whether LSTMs and GRUs train over 1,024 steps. We reuse that quantity, but as a score over many architectures rather than an initialization recipe for a fixed one. Luo et al. use dynamical isometry to make super-network evaluation fair for feedforward image networks with BatchNorm. We drop both the super-network and the BatchNorm dependence, and we add the horizon-matching term, which neither paper has.

The application papers set the targets. CTNAS-EEG is the closest existing work. It runs differentiable search over 1D temporal and 2D spatial convolutions for EEG, and it reports the per-subject gain this proposal tries to deliver cheaply. Its search space has no recurrence, attention or continuous-time blocks, and every subject needs super-network training. ODEBrain models EEG as a continuous-time system with neural ODEs (networks whose hidden state evolves by a learned differential equation, $dz/dt=f_\theta(z,t)$) and shows gains on clinical seizure and abnormality detection. That motivates including ODE blocks in the search space, and the ODEBrain code provides one reference implementation.

Research Program

We first build a mixed temporal search space. Each candidate is a spatial channel-mixing stem followed by two to four temporal blocks. Each block is chosen from dilated 1D convolutions with the kernel sizes CTNAS-EEG found useful, GRU and LSTM cells with two forget-bias settings, causal self-attention, and a small neural-ODE block solved with a fixed-step integrator so that its cost is bounded and comparable. Widths come from a small grid. The space is a few thousand architectures.

Next we build a ground-truth table. We sample about 300 architectures and train each with a fixed recipe and three seeds, per subject, on BCI Competition IV 2a (9 subjects). We train a smaller sample on SEED-IV (15 subjects) and on TUAB. These models are small, so the table is affordable once. Like NAS-Bench-360's NinaPro table, we will release it.

We then compute $S(A)$ and all competing proxies on every architecture and measure rank agreement with trained accuracy. Finally, we run per-subject search: score the full space with $S$, train only the top five, and report final accuracy and wall-clock cost.

Two analyses test the mechanism rather than only the outcome. The first compares $s_A$ at initialization with the same profile after training. If training routinely rewrites the horizon, the score's premise fails, and we would see it here. The second checks whether $d(k)$ differs meaningfully across subjects and tasks. The score can only help per-subject search if the right horizon actually varies between people.

Baselines

The naive reference points are a fixed hand-designed decoder and random search. For the fixed decoder we use EEG-TCNet, which CTNAS-EEG reports at 77.34% on BCI IV 2a (CTNAS-EEG). We retrain it with our pipeline and tuning budget so that splits match. Random search draws five architectures from our space and trains them, which matches the training budget of our "score then train top five" procedure. Beating random search at equal training cost is the minimum requirement for the score to be useful.

The proxy competitors are NASWOT, Synflow, SNIP and EZNAS-A, applied unchanged to our space through their published formulas. EZNAS-A is the strongest of these, at Kendall 0.65 versus 0.57 for NASWOT and Synflow on NAS-Bench-201 CIFAR-10 (EZNAS). We will also evolve an EZNAS/LPZero-style symbolic proxy on our training subjects with the same number of evaluations. This tests whether a general proxy search simply rediscovers our score. To beat these baselines, our score needs a higher Spearman correlation and higher top-10% recall on held-out subjects, with bootstrap confidence intervals that do not overlap. No reported result found on Lacuna for any zero-cost proxy's rank correlation on EEG benchmarks, so these numbers will be new.

The strongest competitor on final accuracy is CTNAS-EEG with per-subject variable structures, which reports 88.52% average on BCI IV 2a (CTNAS-EEG). We will quote that figure. We will also reimplement CTNAS-EEG on our splits and measure its GPU-hours. Success means coming within about two points of the reimplemented CTNAS-EEG at under 5% of its compute. Matching it outright would be a strong result, but it is not required.

A short-training baseline follows budgeted performance estimation, which tunes the epoch and batch budget so that partial-training ranks track full-training ranks. We give it the same wall-clock budget our score uses, plus a budget ten times larger.

Three ablations isolate the key ingredient. The first uses only the isometry term ($\lambda$ term alone), which is Gilboa-style trainability. The second uses only the horizon-matching term. The third replaces $s_A$ with the analytical receptive field, set to infinity for recurrent blocks. If the isometry-only ablation does as well as the full score, the horizon idea adds nothing.

Why not just…?

Why not just use a well-tuned fixed decoder? NAS-Bench-360 found tuned Wide ResNets often beat search on non-vision tasks, so this is a fair worry. In EEG, though, the evidence points the other way: per-subject structures beat EEG-TCNet by about 11 points in CTNAS-EEG. The real question is whether a cheap search keeps that gain. Our retrained EEG-TCNet and the per-subject results answer it directly. If the gap shrinks below a point or two on our splits, we will report that per-subject search is not worth doing here.

Why not just apply NASWOT, Synflow or EZNAS to sequences? These proxies come from one pass and contain no notion of lag. LPZero already documents how they collapse outside convolutional networks. Gilboa et al. show that recurrent trainability depends on products of Jacobians across time, which a single-pass saliency score cannot see. We still run all of them, so if one turns out to work on EEG, the rank tables will show it.

Why not just train each candidate for a few epochs? This is the most serious threat. Short training uses real labels and may simply rank better. Our case rests on cost and on the per-subject setting: a few epochs on a few hundred architectures for every new user is still far more expensive than a few forward and backward passes. We test this head-on with equal-compute and ten-times-compute budgets. If short training wins at equal compute, the contribution narrows. The score would then serve as a pre-filter that decides which candidates get short training, and we would measure that hybrid.

Why not just compute the receptive field analytically? For pure convolutional stacks this could capture most of the horizon signal for free. The receptive field ablation measures exactly that. It cannot tell a GRU with a forget bias of 1 from one with a bias of 5, and it cannot place an ODE block or attention layer on the lag axis. Those are the cases where we expect the Jacobian profile to matter.

Decisive Checks

The cheapest first test uses existing data. NAS-Bench-360 already provides fully trained results for all 15,625 NAS-Bench-201 architectures on NinaPro EMG (NAS-Bench-360). We can compute $S(A)$ and the vision proxies on that table at no training cost. This tests the horizon idea on a temporal biosignal before we build anything. If the score does not beat EZNAS-A there, even for convolution-only architectures, we stop and reconsider.

The main test is held-out-subject rank correlation and top-10% recall on BCI IV 2a, SEED-IV and TUAB, together with per-subject search accuracy against reimplemented CTNAS-EEG at measured GPU-hours.

The mechanism test checks whether the profile at initialization agrees with the trained profile. It also checks whether the data horizon $d(k)$ varies across subjects enough to change which architecture ranks first. If $d(k)$ is essentially the same for every subject, a single searched architecture would serve everyone, and the per-subject argument would rest on spatial or block-type preferences rather than on horizon.