Is a Second Dropout Pass Worth a Second Example? Consistency Training Against Depth-Wise Error Cascades

When a model solves a problem in steps, a wrong intermediate result is usually treated downstream as correct, so errors compound. Accuracy on multi-step tasks therefore falls steeply as the number of steps grows. Two cheap remedies come up again and again. One is simply training on more data. The other is regularized dropout: run each input through the network twice, each time with a different random set of neurons switched off, and penalize the two passes for disagreeing. This proposal asks which remedy actually reduces error cascades, and it measures the answer at equal training compute on reasoning problems generated fresh for each evaluation, so that test items cannot have leaked into training data. Equal compute matters because a second dropout pass costs about as much as reading a second example.

The method has two parts. The first is an evaluation that splits multi-step accuracy into two quantities: how often a single step fails when it is given correct inputs, and how much worse the full chain performs than independent step failures would predict. The second part tests a prediction. More data mainly lowers the single-step error. Consistency training applied at each intermediate step, rather than only at the final answer, mainly lowers the amplification. If that prediction holds, the two remedies are complements rather than substitutes, and we can say how much of each to buy. If it fails, and more data at matched compute does everything consistency does, we learn that too, and practitioners can drop the extra pass.

Core Claim

Existing work on dropout consistency reports gains on single-output or token-level tasks and judges them on static benchmarks. R-Drop applies a bidirectional KL (Kullback–Leibler) penalty between two dropout passes, reporting 30.91 BLEU on WMT14 En-De and a 1.21-point average gain for BERT-base on GLUE. Intra-Distillation extends this to three passes and shows it forces otherwise idle ("lazy") parameters to carry computation. Neither paper asks whether consistency changes how errors propagate across steps. Neither compares the extra forward passes against spending the same compute on more data. Neither evaluates on uncontaminated reasoning, the kind of test on which DyVal found GPT-3.5-Turbo losing 23% arithmetic accuracy from depth level D1 to D4.

Our claim is narrower and testable. Write $A(d)$ for exact-match accuracy on problems that need $d$ dependent steps, and $\varepsilon$ for the per-step error rate measured when the model receives the correct upstream values (teacher-forced). The cascade gap is

$$ G(d) = d \log(1-\varepsilon) - \log A(d), $$

which is zero when steps fail independently, so that $A(d) = (1-\varepsilon)^d$. It is positive when an upstream mistake makes later mistakes more likely.

We predict two things at matched training compute. Scaling data volume lowers $\varepsilon$ but leaves $G(d)$ roughly flat. Step-level consistency regularization lowers $G(d)$ at a given $\varepsilon$. The reason to expect the second effect is that agreement under internal perturbation should transfer partly to robustness against perturbed inputs, which is what a wrong upstream value is.

Research Program

The testbed is a generator in the style of DyVal: random directed acyclic graphs (DAGs, graphs of operations whose edges never loop back) for arithmetic, boolean logic, and reachability, extended with variants inspired by the algorithmic families in BeyondBench. Every test set is freshly sampled with a held-out random seed. Depth and width are controlled independently. Every intermediate node has a known ground-truth value. These properties make both $\varepsilon$ and $G(d)$ measurable and make contamination structurally impossible.

We train decoder-only Transformers from scratch at three sizes, roughly 20M, 100M, and 400M parameters. The model writes one intermediate value per node in topological order. Training from scratch keeps dropout active and controls everything the model has seen.

We compare three training objectives. The first is standard dropout. The second is output-level R-Drop, with the consistency penalty applied only to the final answer token. The third, the proposal's key ingredient, is step-level consistency: the same bidirectional KL penalty applied at the prediction of every intermediate node value, with weight $\alpha$. A three-pass variant following Intra-Distillation is also included.

Data volume is swept over four orders of magnitude, about $10^4$ to $10^7$ problems. We also vary data composition, following the volume decomposition $V = N \times L$ (number of examples times average length) from A Scaling Law for Token Efficiency in LLM Fine-Tuning, which reported 0.278 versus 0.294 BRICC accuracy for "few_long" versus "many_short" subsampling. Here "few_long" means fewer, deeper DAGs and "many_short" means more, shallower ones. For each objective and composition we fit that paper's form, $\text{Accuracy} = A \cdot V^{\beta} + E$, separately to $1-\varepsilon$ and to $G(d)$. The question is whether consistency changes the exponent $\beta$, the floor $E$, or neither.

As a real-data check outside synthetic graphs, we apply the same step-level penalty to the layered nested-entity model of Ju et al.. That model stacks recognition layers from inner entities to outer ones, and its authors name the failure of lower layers as its main limitation. Each layer's tag distribution becomes a "step." We measure whether outer-entity recall conditioned on an inner-layer miss improves.

Baselines

The do-nothing reference is standard dropout trained on the same data stream for the same number of optimizer steps. All objectives share the same tuning budget: an equal number of trials over learning rate, dropout rate, and $\alpha$, selected on a validation set drawn from fresh generator seeds.

The strongest training-time competitor is compute-matched data scaling. Because two-pass consistency roughly doubles forward compute, every consistency run is paired with a standard-dropout run that sees twice as many fresh problems at the same total FLOPs. The proposal beats this baseline only if step-level consistency achieves lower $G(d)$ at $d \geq 8$ than the double-data run, with seed-level confidence intervals that do not overlap. Equal accuracy at the training depths is not enough.

Output-level R-Drop is the ablation that isolates where the penalty is applied. If it matches step-level consistency on $G(d)$, then the intermediate placement is not what helps.

Scheduled noise injection is a direct competitor aimed at cascades. During training, we replace a fraction of teacher-forced intermediate values with plausible wrong values and supervise the downstream steps against the true recomputation given the corrupted input.

At test time, the competitor is self-consistency voting: majority voting over sampled reasoning chains, the simplest relative of the top-$k$ path preservation in BeamAggR. BeamAggR reports an 8.5% average gain over ProbTree across HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle. Voting is run with a sample count chosen so that its total training-plus-inference compute equals that of the consistency model.

For external anchoring, DyVal reports a 23% D1-to-D4 drop for GPT-3.5-Turbo, and BeyondBench reports 59.41% on its Hard suite for GPT-OSS-120B. Neither was obtained under our training regime, so they serve only as context. No reported result was found on Lacuna for R-Drop or any consistency method on DyVal or BeyondBench. For the nested-entity check, the baseline is Ju et al.'s reported F1 of 74.7% on GENIA and 72.2% on ACE2005.

Why not just…?

Why not just train on more data? This is the central competitor, so it is built into the design rather than argued away. Evidence that scale alone falls short on these tasks is mixed. DyVal reports that phi-1.5 and WizardMath-13B scored nearly zero on its tasks despite large pretraining corpora, but those corpora were not structured DAG traces. The compute-matched pairing settles the question directly. If double data closes $G(d)$ as well as consistency does, we report that the extra pass is not worth it.

Why not just apply R-Drop at the output? It is cheaper to implement and is the published method. However, a single final-answer penalty gives no signal about which intermediate step drifted. The output-level ablation tests whether placing the penalty at each step matters. A null result here would narrow the claim to "consistency helps," not "step-level consistency helps."

Why not just train on corrupted intermediates? Scheduled noise injection targets cascades more directly than dropout does, and it is a real threat. It may beat consistency on in-distribution depths. Its weakness is that it only covers the corruption patterns it is shown. We therefore also evaluate on error types held out from the injection distribution, such as off-by-one versus sign errors, and on the combination of injection with consistency. If injection dominates everywhere, we will say so.

Why not just vote at test time? BeamAggR shows that keeping several candidate paths can rescue early mistakes. On the other hand, GeoBench found that o1 localized faulty proof branches only 27.89% of the time against a random-choice rate of 25.71%, which suggests that sampling more paths does not guarantee the model can tell good paths from bad ones. Voting multiplies inference cost on every query, whereas consistency training adds none. The total-compute-matched comparison shows which side of that tradeoff wins at each deployment volume.

Decisive Checks

The prediction is confirmed if two conditions hold together. First, the fitted data exponent for $1-\varepsilon$ is similar across objectives. Second, step-level consistency lowers $G(d)$ at depths beyond those seen in training, relative to the double-data baseline, across all three model sizes. It is refuted if the double-data baseline reaches equal or lower $G(d)$, or if consistency's benefit disappears at the largest data scale. In the latter case, the answer to "is a second pass worth a second example" is "only when examples are scarce," which is itself useful to know. Finally, the nested-entity check confirms the effect beyond synthetic graphs only if consistency raises conditional outer-entity recall after inner-layer misses, not merely aggregate F1.