Two translation models that disagree about a sentence are a good warning that at least one of them is inventing content. Dual-Teacher Agreement for High-Precision Synthetic Data in Low-Resource MT uses this signal to build training data for a small translation model. Two strong multilingual teachers each translate 850,000 monolingual sentences. The method keeps only pairs where the teachers agree and the output is faithful and fluent, about 280,000 of them. The student trained on this filtered set reaches 45.2 chrF++, a character-level translation-quality score. That beats 41.8 for unfiltered synthetic data and 42.7 for real parallel text alone. The catch is cost. Every one of the 850,000 sentences is fully translated twice before anything is thrown away, so about two thirds of a doubled generation bill goes into data that is discarded.
This proposal asks how much of that decision can be made before, or early in, the expensive part: word-by-word decoding by the second teacher. The idea is a staged filter. It spends cheap signals first and full second-teacher decoding last, and only on the sentences where the cheap signals cannot decide. The cheapest signal comes from continual learning, not translation. Select and Distill measures the distance between two teachers' internal representations and uses it to decide which teacher to trust, without generating any output. The goal is to recover the quality of full dual-teacher filtering at close to the cost of running a single teacher.
Core Claim
Most of the keep-or-discard decisions made by dual-teacher agreement filtering can be predicted from signals that cost at most one autoregressive decode. Two such signals are the distance between the two teachers' encodings of the source sentence, and the likelihood the second teacher assigns to the first teacher's translation when forced to read it. A three-stage filter built on these signals should match the student quality of full dual decoding at roughly half the generation compute. At equal compute, it should beat the obvious alternative of running full dual decoding on a random subset of the pool.
A second, smaller claim concerns a weakness the original paper admits: strict surface agreement discards valid paraphrases. Our filter reserves full second-teacher decoding for exactly the uncertain cases. There, it can keep pairs where the teachers differ in wording but agree in meaning. We expect this to help robustness under distribution shift, not only cost.
Research Program
The filter has three stages, ordered by cost.
Stage one runs only the encoders of both teachers on the source sentence. It scores the sentence by how far apart the two teachers' pooled source representations sit once mapped into a shared space. This follows the representation-discrepancy idea of Select and Distill, which does this over an unlabeled reference set to weight distillation. We use it instead to predict whether decoding is worth paying for. Because the two teachers are different models, the mapping between their spaces is a small linear projection. We fit it on a calibration slice of about 20,000 sentences where full dual-teacher decisions are computed once. Sentences scored as near-certain rejects are dropped with no decoding at all.
Stage two decodes the survivors with teacher A only. Teacher B then reads teacher A's output in a single teacher-forced pass, which costs about one forward pass rather than a full generation. The resulting per-token log-likelihood under B, combined with the semantic-similarity score from the original pipeline, sorts candidates into three bins: confident keeps, confident rejects, and ambiguous cases.
Stage three runs full decoding with teacher B only on the ambiguous bin. It then applies the original surface, semantic, and fluency tests, with one change. A pair that fails the surface-agreement test but passes the semantic test with both teachers' outputs is kept, with both translations retained as paraphrase targets.
We account for compute as source tokens encoded plus target tokens decoded, weighted by per-token cost, and report teacher GPU-hours directly. Stage thresholds are chosen on the calibration slice to hit a target generation budget. We analyze how well each stage reproduces the full dual-teacher decisions, measured as precision and recall of kept pairs. We also measure where along the source-length and domain axes the cheap signals fail.
Baselines
The naive reference points come from the original paper and are re-run in our setup. One is training on the 38,000 real parallel sentences alone, reported at 42.7 chrF++. The other is training on all 850,000 unfiltered synthetic pairs, reported at 41.8 chrF++. These tell us whether any synthetic data helps at all.
The strongest existing method is full dual-teacher agreement filtering from Dual-Teacher Agreement for High-Precision Synthetic Data in Low-Resource MT. It reports 45.2 chrF++ in-domain and 41.0 under distribution shift, against 38.1 for the parallel-only baseline. We re-run it with the same two teachers, source pool, student architecture, and fine-tuning schedule. It serves as the quality ceiling at full cost. Beating it on quality is not required. The target is to come within 0.5 chrF++ of it using at most 55% of its teacher GPU-hours.
The fair-compute competitor is the same full dual-teacher filter applied to a random subset of the source pool. The subset is sized so that total generation compute matches ours. This is the baseline to beat: it answers whether smart gating is worth more than simply generating less. Success means a statistically clear gain in chrF++ across three student seeds at equal compute.
A cheaper single-teacher competitor uses teacher A's own sequence log-probability plus the external fluency and semantic filters, with no second model. This tests whether the second teacher is needed at all.
Ablations isolate each stage. We remove stage one, which gives teacher A decoding plus teacher B cross-scoring for everything. We remove stage three, deciding ambiguous cases by cross-score alone. We remove the paraphrase-retention rule, reverting to strict surface agreement. All variants share tuning budgets, with thresholds tuned on the same calibration slice.
Why not just…?
Why not just subsample the monolingual pool and run full dual-teacher filtering on the smaller set? This is the most direct threat, because the original paper already discards two thirds of what it generates. If sentences that pass the filter are spread uniformly and the student saturates below 280,000 pairs, random subsampling could match us. The equal-compute comparison above settles this. We will also plot student chrF++ against retained-pair count for full filtering. If that curve is flat past, say, 150,000 pairs, then subsampling is enough and the gating contribution is small, which we would report.
Why not just filter with one teacher's confidence and a fluency model? The original MT paper reports that single-model fluency filtering misses fluent hallucinations: well-formed output unrelated to the source. That is the motivation for a second model. A related finding cuts the other way. Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs reports that verifying synthetic answers against ground truth did not significantly improve students, which suggests filtering can matter less than expected. The single-teacher baseline measures which effect dominates here.
Why not just cross-score with teacher B and never decode it? This is stage two alone, and it may well be enough. If the ablation without stage three matches the full filter, that is a useful and simpler result, and we will say so. We keep stage three because cross-scoring penalizes valid paraphrases that B would not have produced itself. That penalty is exactly the diversity loss the original paper concedes, so it should show up in the distribution-shift score rather than the in-domain one.
Why not just skip synthesis and translate with a frozen large model at test time? Frozen foundation models are strong when no adaptation is needed. CLIP model is an Efficient Continual Learner reports 75.51% on ImageNet-1K with a frozen model. The purpose of this pipeline, however, is a small deployable student. As TrueTeacher shows, a distilled student can even exceed its teacher: 87.8 against 84.9 ROC-AUC on the TRUE benchmark. We report teacher-direct translation quality as context, not as a competitor under the deployment constraint.
Proposed Experiments
The main testbed reproduces the low-resource translation setup of the dual-teacher agreement paper: 38,000 parallel sentences, 850,000 monolingual sentences, chrF++ in-domain, and the paper's shifted test set. For each method we report student chrF++, teacher GPU-hours, and decoded target tokens. We also report agreement with full dual-teacher keep decisions.
The first decisive check is the equal-compute comparison against random subsampling at 40%, 55%, and 70% of full dual cost.
The second is the error profile of stage one. If encoder discrepancy alone cannot reject at least a quarter of the pool at 95% precision, the continual-learning signal does not transfer to this setting. In that case the method collapses to the cross-scoring cascade, and we report that as a negative result.
The third is the paraphrase test. The rule that keeps pairs with low surface overlap but high semantic agreement must raise the shifted-test score above the 41.0 reported for strict agreement, without lowering in-domain chrF++.
As a secondary check on generality, we apply the same staged logic to generate-then-filter counterfactuals in the style of DISCO. There, a DeBERTa filter rejects candidates after the expensive generator has already run. We ask whether early signals predict rejection there too, using its NLI stress tests. No reported result found on Lacuna for a staged or early-abort variant of either pipeline.
References
- Dual-Teacher Agreement for High-Precision Synthetic Data in Low-Resource MT
- Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
- Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs
- CLIP model is an Efficient Continual Learner
- TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
- DISCO: Distilling Counterfactuals with Large Language Models