Direction-Split Optimizer State for Weight-Tied Bidirectional Sequence Models

Bidirectional sequence models read a time series both forward and backward, so each time step is interpreted with knowledge of what came before and what came after. That second view often matters. In ICU monitoring, a bidirectional LSTM (long short-term memory network, a standard recurrent model) clearly beats a forward-only one, with AUC 0.825 against 0.809, and a supervised variant reaches 0.869 (Zhu et al.). A tighter form of the idea uses the same weights in both directions. Osogami et al. show that alternating forward and time-reversed training on one shared set of weights lets hard-to-estimate connections be learned from the other side. On a sawtooth signal this cuts error tenfold. The same paper reports that this training is exceptionally sensitive to the learning rate. A slightly wrong rate makes the model worse than having no hidden units at all.

We propose that the fragility has a specific and fixable cause. Optimizers such as Adam keep running averages of gradient size and direction. When forward and backward objectives update the same weights, those averages pool two gradient streams that differ in scale and do not share an optimum. The larger stream sets the second-moment estimate and shrinks the smaller stream's effective step. Momentum built up in one direction then leaks into the other direction's updates. Our fix is to keep a separate copy of the optimizer statistics for each direction on the tied weights. The resulting memory equals that of an ordinary untied bidirectional model. The claim is concrete and falsifiable: direction-split state should widen the range of learning rates that train well, and an artificial imbalance injected between the two directions should hurt shared-state Adam but leave split-state Adam largely unaffected.

Core Claim

Let $g_f$ and $g_b$ be the gradients of the forward and time-reversed losses with respect to the shared parameters $\theta$. A shared Adam buffer estimates a single second moment $v \approx \mathbb{E}[g^2]$ over the mixture. The step taken for direction $d$ is then roughly $\eta\, g_d / \sqrt{\tfrac12(g_f^2+g_b^2)}$, where $\eta$ is the base learning rate. Suppose the per-coordinate ratio $r = |g_f|/|g_b|$ is far from 1. The weaker direction's normalized step then shrinks by about a factor of $r$, while the stronger direction's step grows by up to $\sqrt{2}$. That stronger direction is what tends to destabilize training.

A good base learning rate therefore has to suit two directions whose ideal scales differ by a factor of $r$. The usable window narrows by roughly $\log_{10} r$ decades, because the window is the overlap of two windows offset by the scale ratio. Direction-split state normalizes each direction by its own second moment, $\eta\, m_d/\sqrt{v_d}$, where $m_d$ and $v_d$ are the first and second moments for direction $d$. This removes the offset. The prediction comes from the first-order behavior of Adam-style normalization, not from a convergence theorem, so we test it empirically.

The scope matters, and an honest test may shrink it. In untied two-stream models, such as standard BiLSTMs or the bidirectional linear recurrent model BLUR (Liu et al.), each direction already has its own parameters and its own optimizer state. Any directional imbalance there is absorbed by per-parameter or layerwise rules like Fromage (Bernstein et al.), which scales updates relative to each layer's weight norm. Our mechanism is aimed at the case those rules cannot separate: one weight tensor that receives gradients from two time directions.

Relation to Prior Work

Osogami et al. motivate the proposal. They found both the benefit of weight tying and the learning-rate fragility, and they attribute part of it to the time-reversed process not being an exact reversal of the forward one. They propose no remedy beyond manual tuning, and they study only Dynamic Boltzmann Machines, a probabilistic time-series model with hidden units.

Three lines of learning-rate automation supply candidate fixes, and all three treat the training objective as a single stationary target. D-Adaptation (Defazio & Mishchenko) estimates the distance to the optimum from accumulated gradients and matches tuned SGD on ImageNet (76.43% against 76.09%). The parabolic line search PAL (Mutschler & Zell) fits a one-dimensional parabola along the gradient using one extra loss evaluation. The statistical scheduler SASA (Lang et al.) cuts the rate when a stationarity test fires. Under alternating directions, each of these assumptions can break. The two directions may disagree about the distance to the optimum, the parabola may change between steps, and the trajectory may never look stationary. We use them as baselines and as diagnostics, not as the method.

SAGE (Liang et al.) gives each parameter its own rate based on how much the loss depends on it. It shows that shared optimizers under-train some weights, with +1.4 GLUE points on BERT-base. It does not separate the sources of a gradient. Our method separates gradients by source, not by parameter.

Eigenvalue-constrained recurrences such as BLUR, and decay-masked linear attention such as LION (Afzal et al.), stabilize the model's hidden states by design. They leave the outer optimizer untouched, so they are possible confounders that we vary in the experiments.

Research Program

We build weight-tied bidirectional versions of three model families. The first is an LSTM whose single cell runs over both the sequence and its reversal. The second is a linear recurrent unit in the BLUR style with tied recurrence and input matrices and an untied fusion layer. The third is a reimplementation of the Osogami et al. Dynamic Boltzmann Machine, which keeps continuity with the paper that reported the fragility. Each tied model is trained in two ways. In alternating mode, steps alternate between forward and reversed batches, as in Osogami et al. In joint mode, a single step uses the summed loss, computed with two backward passes so that $g_f$ and $g_b$ stay separable.

Direction-split Adam keeps $(m_f, v_f)$ and $(m_b, v_b)$ for every tied tensor. In joint mode it applies the sum of the two normalized updates. In alternating mode it applies only the update for the current direction. Untied parameters keep a single state.

Throughout training we log three diagnostics. The first is the per-tensor gradient ratio $r$. The second is the cosine similarity between $g_f$ and $g_b$. The third is the D-Adaptation distance estimate $d_k$, which shows whether the estimate oscillates when the direction alternates.

The central measurement is the learning-rate window. For each configuration we sweep the base rate over seven decades on a half-decade grid, using three seeds per point. The window is the number of decades over which the seed-median validation score stays within 2% (relative) of the best score that configuration reached. We also report the area under the score-versus-log-rate curve. This summarizes how the configuration performs when the rate is guessed rather than tuned.

Baselines

The do-nothing reference is the tied model trained with standard Adam and one shared state, using the paper's default rate with no tuning. It shows what a practitioner gets out of the box. The tuning-heavy reference is the same model with AdamW, linear warmup and cosine decay, and a full grid over base rate and warmup length. This is the common recipe. Its best grid point sets the ceiling a split-state method must match, and its window sets the floor the method must widen. "Beating" this baseline means a wider window, not only a better peak score.

The strongest competitor on window width is D-Adaptation applied to the tied model with shared state. It is the most established method in the file for removing the base rate, and it is reported to be insensitive to its initial distance guess across 14 orders of magnitude. It is run with its published defaults and the same total step budget. Fromage is the strongest layerwise competitor, run at its default $\eta=0.01$. If either one closes the window gap without direction splitting, our mechanism is unnecessary.

The proposal's key ingredient is isolated by an oracle and an ablation. The oracle runs shared-state Adam with two separately grid-tuned rates, one per direction, using a 2D grid with the same total compute as the 1D grids above. If our hypothesis is right, the oracle should match split-state Adam at its best point, and split-state Adam should get there without the second tuning dimension. The ablation splits only the second moment and shares momentum. This tells us whether the gain comes from scale normalization or from momentum leaking between directions.

For PhysioNet 2012 the published bar is BiLSTM-ST at AUC 0.869, F1 0.793, and precision 0.848 (Zhu et al.). An untied BiLSTM reports AUC 0.825. A tied model with split state that reaches 0.825 or higher across a wider window counts as a positive result. We do not require it to beat 0.869, because that number also reflects step-wise supervision. We will add that supervision as a separate factor.

On ETT, Weather, and ECL, the file gives no reported MSE for BLUR. We will therefore retrain the untied BLUR-style model ourselves under the identical protocol and use it as the reference. For the Osogami datasets, the file records only relative improvements: a "factor of 10" lower RMSE on sawtooth and "up to 90%" lower error on the fuel and temperature data. No absolute values were found on Lacuna for these, so the forward-only and tied baselines will be re-run under our protocol.

Why not just…?

Why not just grid-search the learning rate? On a single model, a grid works. Our window metric directly measures how much grid a practitioner needs. Osogami et al. report that near-miss rates do not just degrade training but invert the benefit of hidden units. A grid is therefore expensive, and it is fragile to anything that shifts the optimal rate, such as a new dataset or a different sequence length. If shared-state Adam's window turns out to be as wide as split-state Adam's, the proposal fails. That is a clean negative result, and we would report it.

Why not just sum the two losses instead of alternating? This is a real threat. Summing removes the step-to-step alternation, which may be where most of the fragility lives. Even under summation, though, the update direction is dominated by whichever gradient is larger, so the weaker direction is still under-trained. Our design crosses alternating and joint modes with shared and split state. If joint mode with shared state already has a window as wide as split state, the contribution narrows to "do not alternate." That would still be a useful finding about the Osogami recipe, but it would be a smaller paper.

Why not just untie the weights and use a layerwise rule like Fromage? Untying gives each direction its own optimizer state for free, and in untied models layerwise rules plausibly handle imbalance. The cost is losing the regularization from tying, which is the source of Osogami et al.'s tenfold sawtooth gain. We compare tied-plus-split against untied-plus-Fromage at matched parameter count and matched optimizer memory. If the untied model wins on both accuracy and window, tying is not worth keeping, and we will say so.

Why not just use D-Adaptation or another learning-rate-free method? These methods assume one objective with one optimum. Under alternation, the logged $d_k$ trace shows directly whether the distance estimate tracks one direction, which would show up as a ratchet toward the larger gradient, or settles. D-Adaptation is also known to need warmup on some architectures. If it widens the window as much as split state, it wins on simplicity, and we would recommend it instead.

Proposed Experiments

The first experiment replicates and measures. On sawtooth, sunspot, fuel price, and NOAA temperature, using the Dynamic Boltzmann Machine and the tied LSTM, we measure the window for shared-state Adam, split-state Adam, D-Adaptation, Fromage, and the two-rate oracle. We report RMSE, the window in decades, and the area under the score-versus-log-rate curve.

The second experiment transfers the comparison to a realistic regime. On PhysioNet 2012 we use the tied BiLSTM, with and without time-weighted step-wise supervision. This supervision weights late steps more heavily. At those steps the backward stream has seen little of the sequence, so we expect it to increase the directional imbalance $r$.

The third experiment tests scale. On ETT, Weather, and ECL across forecasting horizons, we use the tied and untied BLUR-style models. This checks whether eigenvalue constraints already shrink the window gap.

The decisive check is a causal intervention. We multiply the backward loss by a factor $k \in \{1, 3, 10, 30, 100\}$ and track the window for each optimizer. The hypothesis predicts that shared-state Adam loses roughly $\log_{10} k$ decades of window while split-state Adam stays flat. The second check is correlational. Across all models and datasets, the median measured $r$ at initialization should predict the shared-state window deficit. If neither holds, imbalance is not the cause of the fragility, and the method should be dropped regardless of any gains at individual grid points.