Systems that find translated page pairs on multilingual websites mostly work the same way. They embed every page into a shared multilingual vector space and pair pages whose vectors lie close together. Closeness, however, has more than one cause. Two pages from the same section of a site share navigation menus, template text, length, and topic whether or not one translates the other. A raw similarity score therefore mixes the signal we want, translation equivalence, with similarity that comes from shared publishing context. Causal inference has a name for this: confounding, meaning a background variable that drives both which pairs end up compared and how similar they look. This proposal does not stop at relabeling the problem. It derives a concrete scoring rule from that view. A candidate pair is judged by how much more similar it is than non-translated pairs drawn from the same confounder stratum: same template cluster, comparable document lengths, same topical neighborhood. The usual comparison is against the pair's nearest neighbors in general.
In two sentences, the idea is this. Current scoring normalizes similarity against an unconditional background, either by stripping frequent tokens or by subtracting the average similarity to nearby vectors. We instead subtract the similarity expected among known non-translations that match the candidate on pre-translation context, and we use causal reasoning to decide which context variables may be matched on and which must not be. The same rule applies one level down. There it targets the documented tendency of sentence filters to over-select short sentences.
Core Claim
Our claim is that a confounder-conditional null score improves precision on the hard cases that current pipelines still miss. These are same-site pages with shared templates and topics, and true translations whose surrounding layouts differ. The rule needs no new labeled data. Matched controls are free: under the one-to-one alignment constraint, almost every other candidate page on a site is a non-translation, so random within-site candidates form a nearly clean control pool. Contamination is at most one true partner per document. Write $s(a,b)$ for the embedding cosine between pages $a$ and $b$, and $z(a,b)$ for a vector of context features. The adjusted score is
$$\tilde s(a,b) = \frac{s(a,b) - \hat\mu\big(z(a,b)\big)}{\hat\sigma\big(z(a,b)\big)},$$
where $\hat\mu$ and $\hat\sigma$ are the mean and spread of similarity among non-translated within-site pairs whose features match $z$. We estimate them by nearest-neighbor matching in $z$, with a small regression model as fallback.
The causal lens adds a restriction that heuristics lack. The variable $z$ may contain only things that are fixed before and independently of whether $b$ translates $a$. Examples are each page's own length, its share of boilerplate lines, its template cluster, and its topic cluster. Pair-level quantities that translation itself produces must be excluded, such as the length ratio or the overlap of numbers and dates. Adjusting for a consequence of the "treatment" removes real signal, a familiar error in observational studies. This gives a falsifiable prediction: adding those consequence features to $z$ should hurt recall.
Relation to Prior Work
The document alignment literature supplies both the problem and the strongest competitors. Exploiting Sentence Order in Document Alignment found that boilerplate and shared topical vocabulary cause false matches. It countered them with position-aware embeddings over 16 regions, linear inverse document frequency weighting (LIDF) of repeated template text, and re-scoring with the Vecalign sentence aligner. We keep its candidate generation and re-scoring and replace its fixed frequency weights with a matched null. Its fix is unconditional: a "Contact Us" line is down-weighted equally everywhere. Our adjustment depends on which other pages share the candidate's context. CCMatrix mines sentences with a margin criterion, scoring each pair against the average similarity of its $k$ nearest neighbors to fight "hubs", vectors that sit close to many unrelated items. Margin scoring is the special case of our rule where the null is conditioned on vector-space neighborhood and not on publishing context. That makes it the natural ablation. Structural matching in Dirt Cheap Web-Scale Parallel Text from the Common Crawl treats HTML templates as evidence for alignment. We treat template agreement as context that both true and false pairs share, so it should neither reward nor penalize a pair by itself. The cross-domain failures noted in Findings of the WMT 2016 Bilingual Document Alignment Shared Task, where the same content sits on different hosts, are where this difference should matter most.
On the causal side, Text and Causal Inference reviews matching and adjustment with text as the covariate for estimating real-world effects. We borrow its matching-and-evaluation logic and its warning that text representations often lack overlap. The overlap warning motivates our fallback when no matched control exists. Towards Deconfounded Image-Text Matching with Causal Inference is the closest prior use of deconfounding in a pairing task. It applies backdoor adjustment with a hand-built confounder dictionary to image-text retrieval. We replace the learned dictionary with matched controls that are available for free inside each website, and we work across languages, not across modalities. Representation-balancing methods supply a heavier alternative arm. Counterfactual Regression, from Estimating individual treatment effect: generalization bounds and algorithms, learns representations whose distributions match across groups. Representation Learning for Treatment Effect Estimation from Observational Data showed that such global balancing over-smooths, erasing the fine distinctions individual predictions need. That result constrains our design. We expect balancing the encoder across template clusters to blur exactly the fine semantic differences alignment depends on, which is why the core method adjusts scores rather than retraining representations. Finally, the "short sentence trap" reported in Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment motivates the sentence-level arm. There the null is conditioned on each sentence's own length and clause count. The sample-selection framing of Automatic debiased machine learning and sensitivity analysis for sample selection models informs how we will describe the length bias of the output, but we do not use its estimator.
Research Program
The first component is the scorer. Candidates come from a FAISS index over LASER-based position-aware document vectors, following Thompson and Koehn. For each website we sample several thousand random within-site cross-language pairs as controls. We compute their per-document context features: own length, boilerplate fraction (lines repeated on many pages of the site), template cluster from HTML tag-sequence hashing, and topic cluster from monolingual embeddings. We then estimate $\hat\mu,\hat\sigma$ by matching. When a candidate has fewer than a minimum number of controls within a feature-distance threshold, it falls back to the CCMatrix margin. This is an explicit positivity safeguard, and we report how often it fires. The adjusted score then feeds the existing Vecalign re-scoring and the one-to-one assignment unchanged.
The second component is a diagnostic benchmark that exposes confounding directly. WMT16 recall is close to ceiling, so headroom must be made visible. Within the WMT16 English–French domains we stratify test pairs by whether the URL heuristic finds them, since that heuristic misses roughly 40%. We also stratify the strongest competing false candidate by its template and topic overlap with the true partner. We further build a controlled "re-hosted" condition. For true pairs, we remove the shared site-specific boilerplate on one side and replace it with boilerplate from a different domain. This simulates translations published on separate hosts. The gold labels are known, so any recall drop is attributable to template dependence.
The third component is the sentence-level arm. Inside aligned documents, we replace raw sentence-pair scores with the length-conditional null score before selecting the top 5 million words. We then measure both translation quality and the length distribution of what was selected.
Baselines
The naive reference points are URL heuristic matching, which Findings of the WMT 2016 Bilingual Document Alignment Shared Task reports at roughly 60% recall on WMT16, and plain cosine between averaged LASER document embeddings with one-to-one assignment. These show how much confounding raw similarity carries.
The strongest existing method is the Thompson and Koehn pipeline of position-aware embeddings, LIDF weighting, and Vecalign re-scoring. Exploiting Sentence Order in Document Alignment reports 98.5% recall on WMT16 English–French, against a previous best of 96.2%, and a 1.2 BLEU gain on low-resource Sinhala–English ParaCrawl data. We rerun it from its description with identical candidate lists, encoder, and re-scoring, and substitute only the scoring rule. On the full test set, beating it means a statistically reliable reduction in the remaining false-pair errors. Above 98.5% the absolute gain will be small, so we do not claim much there. The substantive bar is a clear recall gain on the URL-unmatchable and re-hosted strata, with no loss on the easy stratum.
The second strongest competitor is CCMatrix-style $k$-nearest-neighbor margin scoring (CCMatrix). We run it on the same candidates with $k$ tuned on the WMT16 training domains under the same budget as our matching thresholds. This comparison isolates the key ingredient: whether the null is conditioned on publishing context or on vector neighborhood.
The third competitor is a discriminative pair classifier in the style of Bicleaner, as used in ParaCrawl. It takes cosine plus the same context features as input and is trained on the 1,624 WMT16 training pairs with hard negatives. It gets the same features and the same tuning budget as our method.
The ablations are as follows. We condition on each feature family alone (length, template, topic) to see which confounder matters. We condition on all of them together. We also add consequence features, the length ratio and number/date overlap, to $z$; the causal argument predicts that this lowers recall. Last, we run a Counterfactual Regression-style variant that fine-tunes the encoder with a Wasserstein balance penalty across template clusters. This tests whether representation balancing adds anything beyond score adjustment or, as the over-smoothing result suggests, costs precision.
For the WMT 2020 Pashto–English and Khmer–English tasks, we compare against the LASER baseline (10.3 BLEU for Pashto, 12.9 for Khmer) and the best submitted systems (12.9 and 15.5 BLEU), all from Findings of the WMT 2020 Shared Task. We use the task's fixed downstream training recipe and the FLORES evaluation sets. Those best systems used mBART pretraining and heavy classifier filtering, so matching them is not expected. The fair comparison is our scorer against margin and LIDF scoring inside the same LASER-based pipeline. Any gain should then be shown to persist when added on top of a strong classifier filter.
Why not just…?
Why not just down-weight boilerplate with LIDF or TF-IDF? This is the most serious threat, because it already produces 98.5% recall (Exploiting Sentence Order in Document Alignment). Frequency weighting removes text repeated across a site. It cannot handle non-translated pages that share non-repeated topical vocabulary, or translations whose context differs. Both are context-conditional effects. The stratified evaluation settles the question. If our gains vanish on the hard strata once LIDF is applied, the matched null is redundant and we will report that.
Why not just use margin scoring? Margin scoring corrects for dense regions of vector space but not for why those regions are dense. A template cluster and a hub may coincide, in which case the two methods give similar results, or they may not. The margin ablation run on identical candidates measures directly whether conditioning on context beats conditioning on neighborhood. We will also report how often the two methods' decisions differ.
Why not just train a classifier with the context features as inputs? In principle a classifier can learn the same adjustment, and it is a real competitor. It has two weaknesses that our experiments test. It needs labels from each new setting. And its labels inherit selection bias: WMT16 gold pairs were found largely with URL matching and Linguee, so they overrepresent easily matched pages. We expect the classifier to do well on the URL-matchable stratum and to degrade on re-hosted pairs. If it does not degrade, the label-free scorer's remaining advantage is only cost, and we will say so.
Why not just use fixed length quotas at the sentence level? Stratified quotas by length bin are a cheap fix for the short-sentence trap. They differ from our rule in one respect: they force a target length distribution, whereas the null score selects pairs that beat their own length stratum without fixing proportions. We run quotas as a baseline under the same 5-million-word budget. If they match on BLEU, the sentence-level arm adds nothing and the contribution rests on the document level.
Decisive Checks
Three outcomes would confirm the proposal. The first is a recall gain on the URL-unmatchable and re-hosted WMT16 strata over both the Thompson and Koehn pipeline and margin scoring, with no loss on easy pairs. The second is the predicted drop when consequence features enter $z$. This is the result that separates a causal design principle from feature engineering. The third is a downstream BLEU gain on Pashto–English and Khmer–English inside a fixed LASER pipeline, accompanied by a selected length distribution closer to that of clean reference bitext. If the first holds but the second does not, the causal framing explains nothing beyond the heuristics and should be dropped. If neither holds, LIDF and margin scoring already capture the confounding that matters in practice. We would report that as a negative result, since it would settle a question this literature has left open.
References
- Exploiting Sentence Order in Document Alignment
- CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB.
- Dirt Cheap Web-Scale Parallel Text from the Common Crawl
- Findings of the WMT 2016 Bilingual Document Alignment Shared Task
- Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates
- Towards Deconfounded Image-Text Matching with Causal Inference
- Estimating individual treatment effect: generalization bounds and algorithms
- Representation Learning for Treatment Effect Estimation from Observational Data.
- Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment
- Automatic debiased machine learning and sensitivity analysis for sample selection models
- ParaCrawl: Web-Scale Acquisition of Parallel Corpora