Many sequence models, such as grammatical error correction (GEC) and machine translation systems, are trained on millions of machine-generated "pseudo" sentence pairs and then fine-tuned on a small amount of real, human-annotated data. These systems often already include a second model that reads the output backwards, from right to left, and uses it to rerank the main model's candidate outputs. We propose using that backward model for a second job: deciding which synthetic training pairs to keep. The argument is simple. By the chain rule of probability, a left-to-right model and a right-to-left model are two factorizations of the same quantity, the probability of the output given the input. If both models fit the real data well, they should give a natural, real-looking pair the same total score. When they disagree sharply, the pair probably lies outside the support of real data, where each model extrapolates in its own way. Simulator artifacts tend to live in exactly those regions.
Concretely, we train a forward and a backward model on the small real seed set that the pipeline already needs for fine-tuning. We score every synthetic pair with both models and drop or down-weight the pairs where the two disagree. We then run the standard pretrain-then-fine-tune recipe on what remains. The method needs no external real-world anchor model, which matters because, as the sim-to-real literature notes, such anchors often do not exist outside natural-image vision. It also reuses a component the strongest GEC and translation systems already train.
Core Claim
Let $s_{\rightarrow}(X,Y)$ and $s_{\leftarrow}(X,Y)$ be the length-normalized log-probabilities that a left-to-right and a right-to-left model, both trained on real data, assign to output $Y$ given input $X$. The directional disagreement is
$$ d(X,Y) = \left| s_{\rightarrow}(X,Y) - s_{\leftarrow}(X,Y) \right|. $$
This quantity would be zero for the true distribution. It should be small on held-out real pairs and large on synthetic pairs that carry generator artifacts.
Our claim is that filtering or reweighting synthetic pretraining data by $d$ improves final real-data accuracy beyond two simpler options. The first is filtering by forward likelihood alone. The second, which is the key control, is filtering by disagreement between two forward models trained with different random seeds. If the two-seed control does as well, then directionality adds nothing beyond ordinary ensemble uncertainty, and the claim is false.
We expect the gain to be largest for artifacts that sit at a particular position in the sequence. Examples are degenerate beam-search endings in back-translated text and formulaic openings, because forward and backward models fail on opposite ends of a sequence. That pattern of opposite failures is the documented reason R2L reranking helps in the first place (English-to-Japanese Diverse Translation by Combining Forward and Backward Outputs). We expect little gain for artifacts spread evenly across tokens, and we will test this limit directly.
Relation to Prior Work
Two GEC and translation papers motivate this work by showing that synthetic data and backward rescoring already coexist without interacting. An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction scaled back-translated pseudo data to 70M sentences. It showed that pretraining on synthetic data and then fine-tuning beats mixing synthetic and real data, which degrades once synthetic data passes about 14M sentences. That paper also used R2L reranking at test time, but it filtered synthetic data only with forward heuristics and a sentence-level error detector. Edinburgh Neural Machine Translation Systems for WMT 16 and NTT Neural Machine Translation Systems at WAT 2019 likewise pair back- or forward-translated synthetic corpora with R2L reranking. In both, the reranker never touches the training data. We keep their recipes fixed and change only which synthetic pairs enter pretraining, so any gain is attributable to the filter.
Theoretically Guaranteed Bidirectional Data Rectification for Robust Sequential Recommendation supplies the closest method. It uses a forward recommender to replace unreliable target items and a backward one to delete unreliable inputs, with error bounds under a noise condition. Its setting, however, is human click noise in real logs, its two directions do separate jobs rather than checking each other, and it involves no synthetic generator. Our contribution is the agreement test between the two directions, applied to synthetic-to-real transfer.
Automated Synthetic-to-Real Generalization constrains the design. It keeps a model from forgetting real-world features while it trains on synthetic images by penalizing divergence from a frozen ImageNet model. As the file notes, that method depends on an external real anchor. Our signal uses only the small real seed set and the model's own two factorizations, so it applies to text, trajectories, and other domains with no large pretrained real model.
Research Program
The main setting is GEC, where the strongest reference system and its numbers are documented. We regenerate synthetic data with back-translation using noisy beam search, following Kiyono et al., at 14M and 70M scales. We train forward and backward transformer models on the real learner seed corpus. We score all synthetic pairs and calibrate $d$ against its distribution on held-out real pairs, so that thresholds are stated as percentiles of real-data disagreement.
We compare three ways to use the score. Hard filtering keeps the pairs below a real-percentile threshold. Soft reweighting multiplies each pair's pretraining loss by a decreasing function of $d$. Token masking uses the per-position version of $d$ to drop the loss only on the region where the two directions disagree. Each filtered corpus then goes through identical pretraining and fine-tuning, and the final system optionally includes R2L reranking.
We also address a question Kiyono et al. left open: does synthetic pretraining degrade the backward reranker differently from the forward generator? We measure each direction's calibration on real data after synthetic pretraining, with and without filtering. A plausible side effect is that the filter matters more for the reranker than for the generator.
The secondary setting is ASPEC English-Japanese with forward-translated synthetic data, following NTT. This checks that the effect is not specific to GEC. It also tests the case NTT raised, where synthetic text may carry direction-specific artifacts.
Baselines
The naive reference point is the full unfiltered synthetic corpus, pretrained and then fine-tuned, with the same architecture, compute budget, and fine-tuning schedule as every other arm. A random filter that keeps the same fraction of data as each disagreement threshold controls for the effect of simply having less data.
The strongest existing method is the full system of Kiyono et al. (An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction). It combines 70M back-translated pretraining pairs, synthetic spelling-error injection, the sentence-level error detector, and R2L reranking, and reports F0.5 of 65.0 on CoNLL-2014 and 70.2 on BEA-2019. We reproduce it under our compute budget and report our reproduced numbers next to the published ones. A comparison counts as a win only against our own reproduction, with the same tuning budget per arm. Beating it means a statistically reliable F0.5 gain over three or more seeds on both test sets with all its components kept on. As a weaker but cleaner success, it also means matching its score with substantially less synthetic data.
For translation, the reference is the NTT recipe (NTT Neural Machine Translation Systems at WAT 2019), which reports 45.83 BLEU on ASPEC English-Japanese. Again, we compare against our reproduction.
Two ablations isolate the key ingredient. The first is two-seed disagreement: two forward models trained on the same real seed with different random seeds, scored the same way. This tests whether directionality matters or whether any second model would do. The second is a forward-likelihood filter: rank synthetic pairs by forward score under the real-trained model, or by the difference between that score and the score under a synthetic-trained model. This tests whether the backward model adds anything. All filters are tuned with the same number of threshold trials on the same development set.
Why not just…?
Why not just rely on sequential pretraining, since fine-tuning already closes the gap? Kiyono et al. show that pretraining then fine-tuning scales monotonically, which suggests the fine-tuning stage already absorbs much of the synthetic noise. That is a genuine threat at 70M pairs. The test is the gap between the unfiltered and filtered arms as a function of synthetic corpus size and real seed size. If the gap vanishes at full scale, the method reduces to a data-efficiency tool, matching accuracy with fewer synthetic pairs and less compute, and we will report it that way.
Why not just filter with a single forward model's likelihood? A real-trained forward model already flags implausible pairs, and this is the most likely competitor. The chain-rule argument says forward likelihood conflates "rare but real" with "off-distribution." Disagreement should separate the two, because both directions agree on rare real pairs. The forward-likelihood ablation decides this. If it matches the disagreement filter, we will conclude the backward model is unnecessary for filtering.
Why not just use an ensemble of forward models to measure uncertainty? Disagreement among models with different seeds is a standard out-of-distribution signal and costs the same. Our bet is that forward and backward models fail in systematically opposite places, so their disagreement is larger and better aimed than disagreement between models trained the same way. The two-seed control at matched compute decides this. If it performs as well, directionality adds nothing.
Why not just use the existing sentence-level error detector? That detector is trained to flag erroneous sentences, not generator artifacts, and it is already part of the strongest baseline. We keep it on in every arm, so any measured gain is on top of it.
Proposed Experiments
The first decisive check is a controlled artifact injection. Into one copy of the synthetic GEC corpus we insert a position-localized artifact: a fixed formulaic suffix on corrected outputs in a known fraction of pairs. Into another copy we insert a uniformly spread artifact: random character substitutions at every position. For each filter, we measure precision and recall at recovering the tainted pairs. We also measure the drop in real-test F0.5 that the artifact causes, and how much of that drop each filter recovers. The prediction is specific. Directional disagreement should beat both forward likelihood and two-seed disagreement on the localized artifact, but not on the uniform one. If it fails on the localized artifact, the mechanism is wrong.
The second check is the main result. It reports F0.5 on CoNLL-2014 and BEA-2019 for all filtering arms at 14M and 70M synthetic pairs and at two real seed sizes, against the reproduced Kiyono system. Each configuration uses three seeds. We also check that the unfiltered model's real-data disagreement correlates with its downstream errors, which tests whether $d$ actually measures what it claims to.
The third check repeats the comparison on ASPEC English-Japanese with forward-translated data, reporting BLEU against the reproduced NTT system. It also includes the calibration analysis of the forward and backward models before and after filtering.
The proposal is supported if directional disagreement beats both forward likelihood and two-seed disagreement on localized artifacts and gives a reliable gain or data saving on at least one real benchmark. It is refuted if the two-seed control matches it everywhere.
References
- English-to-Japanese Diverse Translation by Combining Forward and Backward Outputs
- An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction
- Edinburgh Neural Machine Translation Systems for WMT 16.
- NTT Neural Machine Translation Systems at WAT 2019
- Theoretically Guaranteed Bidirectional Data Rectification for Robust Sequential Recommendation
- Automated Synthetic-to-Real Generalization