Abstract
Cross-lingual document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other. In this paper, we exploit the signals embedded in URLs to label web documents at scale with an average precision of 94.5% across different language pairs. We mine sixty-eight snapshots of the Common Crawl corpus and identify web document pairs that are translations of each other. We release a new web dataset consisting of over 392 million URL pairs from Common Crawl covering documents in 8144 language pairs of which 137 pairs include English. In addition to curating this massive dataset, we introduce baseline methods that leverage cross-lingual representations to identify aligned documents based on their textual content. Finally, we demonstrate the value of this parallel documents dataset through a downstream task of mining parallel sentences and measuring the quality of machine translations from models trained on this mined data. Our objective in releasing this dataset is to foster new research in cross-lingual NLP across a variety of low, medium, and high-resource languages.
Introduction
To train modern machine translation (MT) models or cross-lingual AI architectures like multilingual BERT, researchers require vast amounts of "parallel data"—the same text written in two different languages. Traditionally, this data was sourced from curated, high-quality repositories such as United Nations proceedings or the European Parliament. However, these sources are limited in volume and linguistic diversity, typically covering only a few dozen languages.
The web represents a nearly infinite alternative, but identifying which English page corresponds to which French page is a significant challenge. Manually aligning these documents is impossible at scale due to the quadratic search space. CCAligned addresses this by automating the identification of parallel document pairs within the massive Common Crawl archive, providing a vital resource for over 8,000 language pairs.
Methodology
The researchers employed a two-pronged strategy: a high-precision heuristic for dataset construction and an embedding-based approach for benchmarking.
URL-Based Weak Supervision
The core innovation of this work is the exploitation of predictable URL patterns. Many websites use consistent structures to denote translations, such as changing a language code in the path or subdomain. By stripping these identifiers and matching the remaining URL strings, the authors identified potential pairs with high confidence. This heuristic was applied to 68 Common Crawl snapshots, processing 169 billion documents to find unique pairs.
Content Verification and Embedding Baselines
To ensure the URLs were accurate, a FastText language identifier was used to verify that the document's text matched the expected language. For benchmarking, the authors tested neural embeddings (specifically LASER) to align documents based on text alone. They introduced SLIDF (Sentence Length + Inverse Document Frequency) weighting, which prioritizes long, unique sentences over common "boilerplate" text (like "Click here") to create more distinct and matchable document fingerprints.
Results and Evaluation
The resulting CCAligned dataset contains 392 million URL pairs across 8,144 language pairs. Notably, 292 million of these pairs are between non-English languages, providing a massive boost for low-resource language research.
Human Validation
To assess the quality of the URL-based alignment, human evaluations were conducted across six diverse language pairs representing high, medium, and low-resource categories. The results showed an average adjusted precision ($P_{adj}$) of 94.5%, confirming that URL signals are a robust proxy for content translation.
Downstream Performance
Models trained on CCAligned data achieved BLEU scores (a standard measure of translation quality) comparable to those trained on established corpora like ParaCrawl and significantly outperformed those using WikiMatrix. This demonstrates that the mined web data is of sufficient quality to improve real-world machine translation systems across a wide spectrum of languages.
Conclusion
CCAligned provides a massive, high-quality resource for cross-lingual NLP. While the URL-matching method is high-precision, it is inherently low-recall, missing parallel pages with divergent URL structures. Future work will focus on improving document-level representations to extract even more value from the noisy web, particularly for languages where URL patterns are less standardized.
Summary updated