Many real systems behave mostly linearly, with small, localized nonlinear distortions on top. Examples include vibrating structures, electronic circuits and radio amplifiers. A common way to model them is to split the network into two parallel paths: an explicit linear map plus a nonlinear branch meant to learn only what the linear map misses. Theory says this split protects training: every local minimum is at least as good as the best linear model (Are ResNets Provably Better than Linear Predictors?). But that is only a floor. Nothing in the split stops the nonlinear branch from settling into a useless configuration once the linear path has absorbed most of the loss. Its units may saturate, it may learn linear-looking features that cancel against the linear path, or it may freeze into a small correction. That is premature convergence happening inside one network rather than across a population of search agents.
We propose a simple fix and a test of whether it is needed. At every training step we compute, by least squares on the current batch, the best linear approximation of the nonlinear branch's output in terms of the model's own linear regressors, and we subtract it. The nonlinear branch then cannot represent anything linear. The linear path and the nonlinear path stop competing for the same directions, and the branch's gradient always points at the part of the error no linear map can reach. In the language of the premature-convergence literature, the best linear solution acts as a permanent "already-found solution" that the nonlinear branch is deflated against. Applied within one model and enforced exactly at every step, this differs from three prior approaches: a penalty across sequentially trained models, a per-unit activation transform, and an initialization used only at the first step.
Core Claim
In linear-plus-nonlinear models trained by gradient descent, the most common way to land in a poor local optimum is linear capture. The linear path fits fast, and the nonlinear branch then stalls in a configuration that leaves most of the achievable nonlinear improvement unrealized. We expect differentiable projection of the branch onto the complement of the linear span to cut the rate of such stalls across random seeds. The cut should be larger than what good linear initialization alone, separate learning rates, or a frozen two-stage fit achieve. The comparison should hold most clearly in recurrent state-space models, where the linear and nonlinear parts interact through the state and no static fit separates them. If frozen two-stage fitting or tuned learning rates match the projection, the claim fails, and we will report that.
Concretely, let $z$ be the vector of linear regressors at a time step: the current state, the input and a constant. Let $H$ be the matrix of nonlinear-branch outputs $f_\theta(z)$ over a batch, and $Z$ the matching matrix of regressors. The branch the model actually uses is
$$ \tilde f_\theta = H - Z M^\star, \qquad M^\star = (Z^\top Z + \lambda I)^{-1} Z^\top H, $$
where $M^\star$ is the ridge-regularized least-squares map from regressors to branch outputs, $\lambda$ is a small damping constant, and gradients flow through $M^\star$. The projected branch has zero linear correlation with $z$ on every batch. This is a function-level version of the "zero average slope" condition. The linear matrices $A, B, C, D$ stay fully trainable.
Relation to Prior Work
Two papers motivate the gap. Are ResNets Provably Better than Linear Predictors? proves the "no minimum worse than linear" floor for shortcut architectures, but explicitly does not show convergence to good nonlinear optima, and its guarantee depends on bounded norms. We target exactly the space between that floor and the optimum. Discovering alternative solutions beyond the simplicity bias in recurrent neural networks shows that gradient training of recurrent networks collapses onto the simplest, linearly decodable mechanisms. It escapes them with Iterative Neural Similarity Deflation, which penalizes how well a new network's hidden states can be linearly predicted from those of previously trained networks, restricted to directions the readout ignores. We borrow the deflation idea but change three things. The reference is the model's own linear path rather than a previous network. The constraint is an exact projection rather than a tuned penalty. And it runs inside a single training run, so no sequence of models is needed.
Two papers supply methods we build on or must beat. Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities transforms every hidden unit to zero mean, zero slope and unit variance so that units cannot learn linear trends. This brought plain stochastic gradient descent close to second-order behavior, with 112 MNIST test errors and a learning rate of 1.2 versus 0.45. However, the per-unit variance rescaling sometimes drove units toward zero and made bottleneck decoders collapse. Our projection acts only on the branch output, with no per-unit rescaling, which is designed to keep the decoupling benefit without that failure. Improved Initialization of State-Space Artificial Neural Networks adds a parallel linear state-space model to a nonlinear network, giving the generalized residual state-space network (gR-SS-NN). It initializes the linear matrices with the Best Linear Approximation (BLA, the linear model fitted directly to input-output data). On Bouc-Wen this sharply reduces variance across runs. The initialization is used once, at step zero, and nothing afterwards stops the linear and nonlinear parts from re-entangling. Our projection keeps the separation active throughout training, and it can be combined with BLA initialization.
Other work is context rather than competition. Residual Policy Learning shows that a fixed base controller plus a learned residual escapes a 0%-success optimum on sparse-reward Hook tasks, reaching about 80% success. Multiple-Frequencies Population-Based Training shows that greedy short-horizon selection favors fast-converging configurations and causes collapse. We read the linear path as the internal "fast agent" that wins early, in the same spirit as that population-level finding. Population methods are expensive, though, and treat the network as a black box. Our intervention costs one small least-squares solve per batch.
Research Program
The work has three parts. The first is diagnosis. On a synthetic teacher, a linear dynamical system plus a few localized nonlinear bumps whose amplitude and support we control, the global optimum is known. That lets us measure entrapment directly as the fraction of seeds that end with simulation error more than twice the teacher-achievable error. Alongside the error, we track during training the "nonlinear share": the fraction of output variance carried by the part of the nonlinear branch that is orthogonal to the linear regressors. We also track the fraction of saturated branch units and the norm of the branch's linear component. The hypothesis predicts a specific signature in stalled runs. The nonlinear share should plateau early, and the branch's linear component should grow and cancel against the linear path.
The second part is the method itself. We implement projected training for gR-SS-NN-style models. For a minibatch of simulated subsequences, we stack regressors and branch outputs along each trajectory, apply the projection above, and backpropagate through both the projection and the recurrence. A layer-wise variant for feedforward networks projects each decoupled block's branch against that block's input. This variant lets us compare against the per-unit transforms of the LTMLP paper on its original MNIST setting.
The third part is analysis. For static regression with a single linear layer, projection makes the loss split exactly into a linear term and a nonlinear-residual term, which removes the flat valley in which the linear path and the branch trade off the same function. We will state this precisely. We will also check empirically whether the curvature spectrum flattens, as reported for per-unit transforms. For the recurrent case no exact split exists, and we treat that case empirically.
Baselines
The naive reference point is a plain state-space network with random initialization and no parallel linear path, trained by Adam. This is the configuration that Improved Initialization of State-Space Artificial Neural Networks reports as high-variance and frequently stuck. It shows how often entrapment happens with no remedy.
The strongest existing method is gR-SS-NN with BLA initialization from the same paper. On Bouc-Wen it reports a simulation RMSE of $7.14 \times 10^{-6}$, against a previous best of $1.2 \times 10^{-5}$. For Wiener-Hammerstein we found no reported result on Lacuna for gR-SS-NN, so we will reproduce it ourselves. We will reimplement the method with the same network sizes, the same data splits and the same tanh activations, and confirm that we approximately recover the reported Bouc-Wen RMSE before running any comparison. Every method then gets the same number of gradient steps, the same optimizer, and the same hyperparameter search budget over learning rate and width: a fixed number of trials, scored on validation RMSE. Results are averaged over at least 50 seeds. Beating this baseline means one of two outcomes at no worse median test RMSE. The first is a statistically significant reduction in the entrapment rate and the across-seed spread. The second is a lower median RMSE at equal entrapment rate.
The two-stage fit freezes the linear matrices at the BLA and trains only the nonlinear branch on what remains. This tests whether simply removing the linear path's ability to move is enough.
The two-learning-rate baseline tunes separate step sizes for the linear and nonlinear parameters within the same search budget. This tests whether the problem is only one of timescales.
The per-unit-transform baseline applies the LTMLP zero-mean, zero-slope, unit-variance transformation from Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities inside the nonlinear branch.
The restarts-at-matched-compute baseline runs the plain gR-SS-NN with more seeds and keeps the best one by validation score, using the same total compute.
Two ablations isolate the key ingredient. The first applies the projection with random rather than BLA initialization, which asks whether projection substitutes for good initialization. The second replaces the hard projection with a soft penalty on the branch's linear predictability, in the style of INSD, whose coefficient must be tuned. This asks whether exact enforcement matters.
Why not just…?
Why not just use BLA initialization, which already fixes entrapment? This is the most serious threat. The reported variance reduction on Bouc-Wen is large. Our hypothesis is that initialization helps at step zero but leaves re-entanglement possible later, which should show up as nonlinear share falling or plateauing mid-training in some seeds. If BLA-initialized runs show no such late drift and no residual entrapment on the synthetic teacher, then projection adds nothing in this regime, and we would narrow the claim to random or poor initializations. Those still matter whenever the BLA is badly biased, as it is for strong nonlinearities. The random-initialization ablation measures this directly.
Why not just freeze the linear part and fit the residual? For static regression this is nearly equivalent to projection, and we expect a tie there. A tie would honestly confirm our own decomposition argument. In a state-space model, though, the nonlinear term changes the state trajectory. The best linear matrices given the nonlinear branch then differ from the data-fitted BLA, and freezing them locks in a biased linear model. Projection lets the linear matrices keep adapting. The decisive comparison is two-stage fitting versus projection on Bouc-Wen and on synthetic teachers with increasing nonlinear strength. Projection should win as strength grows. If it does not, the two-stage fit is the better recommendation because it is simpler.
Why not just tune separate learning rates, or rely on Adam? Adaptive optimizers rescale coordinates individually. They cannot remove a redundancy that lives in function space, where the linear path and the branch's linear component represent the same map. The LTMLP results show that architectural changes give gains that ordinary first-order training did not reach. Still, a well-tuned two-rate schedule could close much of the gap. It receives the same search budget as every other method, and we will report its entrapment rate side by side.
Why not just run more random restarts? Restarts reduce the chance that the best run is stuck, but they multiply compute. The MF-PBT paper shows that broad search, here random search, can also lose to targeted mechanisms: its random search reached an interquartile-mean Humanoid reward of 17,713 against 23,793 for MF-PBT. Our matched-compute restart baseline tests whether projection with fewer runs beats the best of many plain runs.
Proposed Experiments
The decisive test runs on Bouc-Wen and Wiener-Hammerstein, with the synthetic teacher as a controlled companion. The prediction is that projected gR-SS-NN lowers the entrapment rate and the across-seed spread of test RMSE relative to BLA-initialized gR-SS-NN at matched tuning and compute, while matching or improving median RMSE relative to the reproduced $7.14 \times 10^{-6}$ on Bouc-Wen. The diagnostic prediction is that stalled baseline runs show an early nonlinear-share plateau and growing cancellation between the branch's linear component and the linear path, and that projected runs show neither. On the synthetic teacher we sweep nonlinear strength. The prediction is that projection's advantage over the two-stage fit grows with strength, while its advantage over BLA initialization is largest under random initialization.
The secondary test takes the layer-wise variant to MNIST against the LTMLP setting, which reported 112 errors. We also run a bottleneck autoencoder there, where per-unit transforms were reported to collapse. Projection should match LTMLP's tolerance for larger learning rates without the collapse. If projection collapses there too, the unit-variance transform was not the cause, and we will report that.
The outcome that would kill the idea is any of the simpler baselines matching projection on entrapment rate across the nonlinear-strength sweep: two-stage fitting, tuned two-rate training, or matched-compute restarts. In that case we would conclude that linear capture is not the bottleneck it appears to be.
References
- Are ResNets Provably Better than Linear Predictors?
- Discovering alternative solutions beyond the simplicity bias in recurrent neural networks
- Pushing Stochastic Gradient towards Second-Order Methods - Backpropagation Learning with Transformations in Nonlinearities
- Improved Initialization of State-Space Artificial Neural Networks
- Residual Policy Learning
- Multiple-Frequencies Population-Based Training