When the same web page exists in many languages, current pipelines for building translation training data still treat it as a set of separate two-language problems. Each translation is aligned sentence by sentence against English, and pairs between other languages are inferred through that English "bridge." Medical imaging met this problem decades ago. If every brain scan is warped onto one arbitrarily chosen subject, every result inherits that subject's quirks. The field's fix was unbiased groupwise registration: align the whole group at once to a shared, implicit average that no single member defines.
We propose to carry this idea over to text. All $N$ language versions of a document are aligned jointly to a learned "consensus document." This is a sequence of embedding slots, estimated as a robust average over the versions. A sentence is judged noisy when it sits far from its slot relative to how the rest of the group behaves, not when it disagrees with English.
The core claim is concrete. A reference-free consensus alignment, built by iterative averaging with explicit insertion and deletion states, should match pairwise pivoting on clean data. It should clearly beat pivoting when the bridge language is itself corrupted or partially translated. It should also flag noise without the clean-data helper translation models that current filters need. The skeptic's strongest alternative is to run all pairwise alignments and then reconcile them by voting. That is a real competitor, and the experiments are designed so it could win.
Why this gap exists
Several lines of work motivate the proposal. DocHPLT builds 124 million web document pairs across 50 languages, all paired with English. It obtains its non-English pairs only by pivoting through English, so any omission in the English document propagates to the derived pair. The same paper finds that training on full web documents degrades translation relative to 10-sentence chunks, which it attributes to accumulated alignment errors.
From Unaligned to Aligned shows that multi-way parallel data helps multilingual language models. Its TED2025 corpus, however, sidesteps alignment entirely by using human video timestamps. It also reports that using English as a pivot is task-dependent, which leaves pivot-free automatic alignment of noisy text unaddressed.
The bitext-cleaning tools constrain the design and are all pairwise:
-
Vecalign aligns exactly two documents in linear time. It uses coarse-to-fine dynamic programming over averaged sentence embeddings, and its block-averaging trick is something we reuse. - Dual conditional cross-entropy filtering scores a pair by the agreement of two inverse translation models. Those models require clean bitext, which low-resource languages lack.
-
BitextEdit repairs the "weaker" side of a pair toward the "stronger" side. This writes the stronger side's errors into the repaired sentence.
The method itself comes from groupwise registration:
- The iterative-bootstrapping atlases of Zhang et al. and Chintalapani et al. repeatedly re-estimate a population average and re-align every subject to it. We take this loop directly.
- Temporal Prototype Learning synchronizes many videos to a shared one-dimensional prototype with an inverse-consistency constraint. This is the closest sequence analogue.
That paper's smooth, continuous time warps cannot represent what web text actually does: drop sentences, insert boilerplate, and merge or split sentences. Our alignment therefore replaces continuous warps with discrete dynamic programming that has explicit skip states. Entropy-based groupwise objectives such as Rényi entropy point-set registration (Group-Wise Point-Set Registration Based on Rényi's Second Order Entropy) motivate scoring agreement across the whole ensemble rather than pair by pair.
Core Claim and Method
Each language version $i$ of a document is a sequence of multilingual sentence embeddings, in the LASER style used by Vecalign. The consensus document is a sequence of $K$ slot vectors $T_1,\dots,T_K$. Each version is aligned to the consensus by a monotone alignment $a_i$. The alignment may map a block of one to three consecutive sentences to a slot, represented by their averaged embedding as in Vecalign. It may also leave a slot empty (a deletion in that language) or leave a sentence unassigned (an insertion, such as boilerplate). We minimize
$$\sum_{i=1}^{N}\sum_{k:\,a_i(k)\neq\varnothing}\rho\big(1-\cos(T_k,\,\bar e_{i,a_i(k)})\big)\;+\;\lambda\,(\#\text{skips in }a_i),$$
where $\bar e_{i,a_i(k)}$ is the averaged embedding of the block that version $i$ assigns to slot $k$. The function $\rho$ is a robust loss (Huber or truncated) that keeps one corrupted language from dragging a slot. The weight $\lambda$ prices insertions and deletions.
Optimization alternates two steps, as in atlas bootstrapping. With the slots fixed, each version is aligned independently by windowed dynamic programming. With the alignments fixed, each slot is re-estimated as a trimmed mean of the blocks assigned to it. Slots supported by too few languages are pruned, and slots whose neighbours are poorly fit are split.
Three choices keep the procedure free of reference bias:
- Initialization. The consensus starts from the average of all versions after resampling them to a common length. No single document is used as the seed.
- Centering. A centering step requires the average alignment offset across languages to be zero at every slot. This is the discrete counterpart of the inverse-consistency constraint in Temporal Prototype Learning.
- Per-language offset removal. Each language's mean embedding is subtracted before averaging. This counters the language-specific offsets and "hubness" that Vecalign handles with ad hoc normalization.
The result yields two outputs.
- Many-way alignment. Every pair of languages can be read off through shared slots, with no bridge language. The cost per iteration is linear in $N$, whereas all-pairs alignment is quadratic.
- Noise score. For each sentence we compute its robust distance to its slot, standardized by the spread of the other languages at that slot. An unassigned sentence counts as an insertion. A low-resource sentence is then judged against several high-resource translations at once, and no translation model is needed. This is how the proposal tries to break the circularity behind dual cross-entropy filtering.
As a secondary, clearly separate step, flagged partial translations can be handed to a BitextEdit-style editor conditioned on all other versions of that slot rather than on one "stronger" side.
Research Program
Data. First we build evaluation testbeds with gold many-way alignment and controlled noise.
- TED2025 with timestamps removed. We take TED2025 talks, discard the timestamps, and corrupt the transcripts with web-like noise. The noise types follow the taxonomy in the noisy-corpora literature: sentence deletion, local reordering, inserted boilerplate, truncated partial translations, and untranslated source segments. Corruption is applied at varying rates, sometimes to one language only and sometimes to several.
- The Bible corpus. We use the multilingual corpus Vecalign evaluates on, with verse markers stripped.
- Real web groups. From DocHPLT we collect English documents that are paired with several non-English documents. Each such set is treated as a group, and our method realigns its sentences jointly, ignoring which side was English.
Analyses. Beyond accuracy, we measure bias directly. For any pivot-based method, alignment quality between languages $j$ and $l$ depends on the choice of pivot language, and we report the variance of F1 over pivot choices. Our method has no pivot, so it should show near-zero variance by construction. We also check that it is not merely averaging away low-resource languages. To do so, we report F1 separately for each language's resource level.
Baselines
Naive reference point. This is the DocHPLT-style pipeline: Vecalign between each language and English, with non-English pairs composed through English, followed by standard LASER-similarity filtering at a fixed threshold. It is what large web corpora are built with today, so any gain must be measured against it. A further "do nothing" reference is the raw mined pairs with no filtering at all.
Strongest competitor: pairwise synchronization. Vecalign is run on all $N(N-1)/2$ language pairs. The resulting alignments are then made mutually consistent by cycle-consistency voting, keeping sentence correspondences that survive transitive agreement across triangles of languages. This is transformation synchronization in the sense used for multi-view point-set registration, for example gravitational alignment. It uses the same multi-way evidence our method uses, without a learned consensus, so it is the right test of whether the barycenter adds anything.
Strongest filter where helper models exist. We run dual conditional cross-entropy on high-resource pairs, using helper models trained on the same clean data available to every method.
Fairness and tuning. All methods use identical embeddings, identical candidate documents, and the same budget of tuning trials on a held-out development split for thresholds, window size, and $\lambda$. Wall-clock time is reported alongside quality.
Reported numbers. On the Bible corpus, Vecalign "improved F1 scores by an average of 28 points over Hunalign," and on Text+Berg it "outperformed the previous SOTA by 5 F1 points." The file gives these only as relative gains. No reported result was found on Lacuna for absolute many-way alignment F1 or for pivot-free non-English translation on FLORES-200. We will therefore rerun the baselines ourselves.
What counts as beating them. Our method must equal the pivot pipeline on clean data. It must exceed it on non-English pairs when the English version is corrupted. Against the synchronization baseline, it must match or exceed alignment F1 at substantially lower cost when $N \ge 10$.
Ablations. These isolate the key ingredient:
- English-seeded consensus. The consensus is initialized from the English document and not centered. This keeps the iterative machinery but reintroduces reference bias.
- Plain mean. The robust trimmed mean is replaced with an ordinary mean.
- No skip states. The insertion and deletion states are removed.
Why not just…?
Why not just keep pivoting through English and filter harder? Pivoting is cheap and English alignments are usually the most accurate. The DocHPLT authors note, however, that errors in the bridge document propagate to every derived pair. Our English-only corruption experiment tests exactly this. If pivot quality on non-English pairs does not degrade more than ours as English noise rises, the bias argument fails. In that case we would report the method as offering no advantage over pivoting.
Why not just run all pairwise alignments and vote for consistency? This is the real threat. It uses the same multi-way signal, and with enough languages, voting may recover nearly everything a consensus does. Our expected advantages are three:
- cost that grows linearly rather than quadratically in the number of languages;
- an explicit per-slot reference, which makes the noise score well defined for a single sentence;
- better behaviour when most pairs are partially corrupted, since voting then has few clean triangles to rely on.
The noise-rate sweep against this baseline settles the question. If synchronization matches us at every noise level, the honest conclusion is that the contribution reduces to efficiency and the noise score. The paper would be framed that way.
Why not just use translation-model filters or BitextEdit? Both need clean bilingual data or a trained bilingual editor. That is precisely what is missing for the low-resource languages where noise is worst. BitextEdit's own gains (5–8 BLEU on English–Occitan) come from repairing toward one side assumed to be correct. We compare on its high-resource settings, where helper models exist. In the low-resource settings it cannot be initialized without the clean data that is missing there, so we only note this limitation.
Proposed Experiments
Alignment accuracy. On TED2025 with timestamps removed and on the stripped Bible corpus, we report many-way and per-pair alignment F1 as a function of noise rate and of which languages are corrupted. Pivot-choice variance is reported alongside.
Noise detection. We report precision and recall for the injected insertion, deletion, and partial-translation errors.
Translation downstream. Non-English translation models are trained on pairs extracted from DocHPLT groups by each method, then evaluated with BLEU and chrF on FLORES-200 directions not involving English.
Language-model downstream. We rebuild a multi-way TED2025 training set from the corrupted transcripts with each method and repeat the continued-pretraining comparison. From Unaligned to Aligned reports MMMLU accuracy of 22.48/41.38 (low/high-resource) for gold multi-way data versus 19.64/36.26 for unaligned data. How much of that gap each automatic method recovers is the cleanest measure of whether reference-free alignment is worth it.
Decisive checks. Two results decide the project. The first is the English-only corruption sweep against the pivot pipeline. The second is the full noise sweep against all-pairs synchronization.
References
- DocHPLT
- From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora
- Vecalign: Improved Sentence Alignment in Linear Time and Space
- Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora
- BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation
- Unbiased White Matter Atlas Construction Using Diffusion Tensor Images
- Statistical Atlases of Bone Anatomy: Construction, Iterative Improvement and Validation
- Synchronization of Multiple Videos
- Group-Wise Point-Set Registration Based on Rényi's Second Order Entropy
- Fast Simultaneous Gravitational Alignment of Multiple Point Sets