Comparing Learning Speed Across Curved and Flat Streams in EEG Decoders

Many brain-signal decoders now combine two kinds of input stream. One treats the recording as a set of channel covariance matrices: symmetric positive-definite (SPD) matrices that describe how electrodes co-vary. These matrices live on a curved space, so the stream is processed with Riemannian geometry and constrained weights. The other stream is an ordinary flat network over spectral or temporal features. Multimodal learning has a well-documented failure: the stream that is easier to learn from captures training early, and the other stream's encoder stops improving. The fixes that work best detect which stream is "learning faster" during an early critical period and slow it down. The detectors use parameter-space quantities such as gradient norms, gradient cosines, and Fisher-information traces. We argue that these quantities cannot be compared across a stream whose weights sit on a curved, orthogonality-constrained space and a stream with unconstrained Euclidean weights. Their values depend on how each stream happens to be parameterized, not on how much it is learning.

The proposal measures each stream's learning speed in the space of predictions instead. The measure is how much the model's output distribution changes when only that stream's parameters take their next step. That quantity is invariant to reparameterization by construction. We use it to drive an early-window regulator for hybrid Riemannian–Euclidean EEG decoders. The study is designed so that it can fail cleanly. If the cheaper Euclidean detectors make the same throttling decisions, or if these decoders show no suppression at all, the experiments will show it.

Core Claim

Two hybrid EEG architectures already observe that streams compete without doing anything about it. RFNet and the closely related spatio-temporal Riemannian/Euclidean model each pair a tangent-space covariance stream with an LSTM over differential-entropy and power-spectrum features. Both fuse the streams with soft attention. Both report that the spatial stream dominates motor imagery while the temporal stream dominates emotion and vigilance decoding. That is evidence of strong asymmetry. It is not yet evidence that the weaker stream is suppressed during joint training. We take it as motivation, not proof.

On the balancing side, InfoReg supplies the method we build on. It tracks the trace of each encoder's Fisher information, identifies an early "prime learning window" after which a starved modality cannot recover, and penalizes the dominant encoder only inside that window. On CREMA-D this revived the video branch's standalone accuracy from 35.14% to 49.65%. MASAM uses a related signal: cosine alignment between each modality's gradient and the joint gradient. Both are parameter-space signals designed for Euclidean encoders. In a hybrid decoder whose spatial stream uses BiMap weights on the Stiefel manifold, as in EE(G)-SPDNet, the Euclidean gradient carries components normal to the constraint that never move the weights. The weights' scale is also fixed by orthogonality, whereas the LSTM's weight scale is free. A Fisher-trace ratio between the two streams therefore partly reflects parameterization.

Our claim is twofold. First, a function-space step signal makes stream comparisons in hybrid geometric decoders meaningful. Second, early-window regulation driven by that signal reduces stream suppression and improves accuracy compared with the same regulator driven by Euclidean signals. For stream $m$ with proposed update $\Delta_m$, applied through a retraction for Stiefel weights so that the update stays on the constraint set, define

$$s_m = \frac{1}{|B|}\sum_{x \in B} \mathrm{KL}\big(p_\theta(y\mid x)\,\|\,p_{\theta \oplus \Delta_m}(y\mid x)\big) \approx \tfrac{1}{2}\,\Delta_m^\top F_m \Delta_m .$$

Here $B$ is a minibatch, $p_\theta$ is the fused model's predictive distribution, $\theta \oplus \Delta_m$ means updating only stream $m$, and $F_m$ is that stream's Fisher block. Because the left-hand side depends only on outputs, $s_m$ is unchanged by any reparameterization that leaves the network's function fixed. That covers rescaling a layer and compensating in the next one, and it covers swapping Euclidean for Stiefel coordinates. The regulator then uses $s_m$ in place of InfoReg's Fisher-trace change rate.

The geometric adaptations of the update itself are standard and not claimed as new. They consist of scaling the Riemannian gradient before retraction, which keeps BiMap weights orthogonal, and keeping SPD activations positive definite. The contribution is the comparison signal and the evidence about whether it matters.

Research Program

The work has three parts, and each can stop the project.

The first part is a diagnosis: does stream suppression actually occur in hybrid EEG decoders? We train a hybrid model whose spatial stream is learnable end to end. We use EE(G)-SPDNet's convolutional filterbank, covariance pooling, and BiMap/ReEig/LogEig layers, rather than RFNet's mostly fixed tangent features, because fixed features have little to starve. The temporal stream is RFNet's LSTM over DE/PSD features. After joint training, we freeze each encoder and fit a linear probe on it. We compare that probe's accuracy with the same encoder trained alone, which is the protocol behind InfoReg's 35.14% versus 49.65% video result. If neither stream's probe falls meaningfully below its unimodal counterpart on SEED and BCI-IV 2a, balancing is unnecessary here and we report that negative result.

The second part is an invariance analysis. We build pairs of networks that compute identical functions but are parameterized differently. In one pair, an LSTM layer is rescaled and the next layer compensates. In another, the BiMap weights are written in unconstrained versus Stiefel coordinates. We log each detector across training on both members of each pair. The detectors are the Euclidean Fisher trace, the Riemannian-gradient Fisher trace, MASAM-style cosine alignment, OGM's confidence ratio, and $s_m$. We also record whether each detector would trigger throttling, and which stream it would throttle. The prediction is that $s_m$ and OGM's output-based ratio make identical decisions within each pair, while the parameter-space detectors do not. A second stress test applies random per-subject amplitude rescaling during cross-subject training and measures how much each detector's decisions shift.

The third part is the regulator. It is InfoReg's early-window penalty, with the window and the dominant stream chosen by $s_m$. The penalty acts on Riemannian gradients for constrained weights. $s_m$ is computed every $k$ steps, and only during the early window, at a cost of one extra forward pass per stream.

Baselines

The naive reference is joint training of the hybrid model with concatenation fusion and no balancing, together with each stream trained alone. The second reference is RFNet-style soft-attention fusion. For it we cite the reported numbers: 93.72% accuracy on SEED, kappa 0.6734 on BCI-IV 2a, and RMSE 0.0348 with Pearson correlation 0.9890 on SEED-VIG (RFNet). We also rerun it in our pipeline with our splits.

The strongest competitor is InfoReg applied unchanged, using the Euclidean Fisher trace. It is the closest method in mechanism, and it is the one whose early-window premise we inherit. No reported result was found on Lacuna for InfoReg or any balancing method on EEG. Its reported anchor is 71.90% on CREMA-D, against 66.61% for joint training (InfoReg).

We also run OGM-GE (OGM-GE). It scales each branch's gradient by $1-\tanh(\alpha\rho)$, where $\rho$ is a confidence ratio, and its signal is already output-based. MASAM (MASAM) runs under a matched wall-clock budget, since it needs extra backward passes.

All methods share the same architecture, splits, number of epochs, and hyperparameter search budget, for example the same number of trials over penalty strength and window length. Each run uses five seeds, with subject-wise and session-wise splits on the BCI data. Beating InfoReg means a statistically reliable gain in accuracy or kappa on the dataset where the probes showed suppression. It also requires a smaller gap between the lagging stream's probe accuracy and its unimodal accuracy.

The key ablation holds the regulator fixed and swaps only the detector: Euclidean Fisher trace, Riemannian-gradient Fisher trace, cosine alignment, and $s_m$. A second ablation removes the early-window restriction.

Why not just…?

Why not just run InfoReg or OGM-GE unchanged on the hybrid model? This is the real threat. If the two streams' parameter scales happen to be comparable on these datasets, Euclidean InfoReg may make the same throttling decisions, and the geometry adds nothing. The detector-swap ablation settles this directly. If the Euclidean and function-space detectors trigger on the same stream, in the same window, with the same downstream accuracy, we report that the function-space signal is unnecessary in practice. OGM-GE is the subtler threat, because its confidence ratio is also invariant. It differs in measuring which branch is currently more confident rather than which is learning faster, and in acting throughout training. InfoReg's evidence that suppression is set early is why we expect rate inside the window to matter more. The OGM-GE comparison tests that expectation.

Why not just weight the losses statically, or add unimodal auxiliary heads? InfoReg reports that suppression is concentrated in a transient early window. A single fixed weight either under-protects the weaker stream early or over-penalizes the stronger one late. Even so, a tuned static weight with auxiliary heads is cheap, and it might match dynamic methods on small EEG datasets. It receives the same tuning budget, and if it ties, that is the finding.

Why not train the streams separately and fuse their predictions, or pretrain them separately and then fine-tune jointly? Separate training removes gradient competition entirely, so it is the cleanest test of whether joint training is buying anything. It does give up cross-stream interactions. The covariance between frequency bands that EE(G)-SPDNet found useful only exists inside a joint model. Whether that matters across modalities is an empirical question. Both the late ensemble and the pretrain-then-fine-tune pipeline are included as baselines. If either matches the regulator, active balancing is not justified for these decoders.

Proposed Experiments

The main benchmarks are SEED, where the temporal stream is expected to dominate, and BCI-IV 2a, where the spatial stream is expected to dominate. They give opposite dominance directions and so test whether the detector picks the correct stream without being told. SEED-VIG adds a regression check. All three are evaluated with the metrics reported in the spatio-temporal model. The Riemannian components reuse GitZH-Chen/SPDMLR and rkobler/TSMNet. Cross-subject stress tests use the 109-subject PhysioNet motor imagery data used by Federated Transfer Learning for EEG.

As a control, we run the function-space regulator on CREMA-D inside the BalanceBenchmark toolkit (BalanceBenchmark). There both encoders are Euclidean, so it should roughly match InfoReg's reported 71.90%. A large drop there would mean the signal is worse than the Fisher trace, independent of geometry.

There are three decisive checks. First, the probe diagnostic must show suppression on at least one EEG task; otherwise the project ends as a negative result about EEG hybrids. Second, the function-identical network pairs must show that the parameter-space detectors change their decisions while $s_m$ does not; otherwise the invariance argument is moot. Third, when we swap only the detector inside the same regulator, the function-space signal must beat the Euclidean Fisher trace on the suppressed task. It must also close more of the gap between the lagging stream's probe and unimodal accuracies, at no more than one extra forward pass per stream during the window. If the first two checks pass but the third does not, invariance is real but irrelevant to accuracy. That is still worth knowing, but it is a narrower paper than the one proposed.