A General-Purpose Multilingual Document Encoder

Onur Galoğlu Robert Litschko, Goran Glavašš2023 · DOI

Abstract

Massively multilingual pretrained transformers (MMTs) have tremendously pushed the state of the art on multilingual NLP and cross-lingual transfer of NLP models in particular. While a large body of work leveraged MMTs to mine parallel data and induce bilingual document embeddings, much less effort has been devoted to training general-purpose (massively) multilingual document encoder that can be used for both supervised and unsupervised document-level tasks. In this work, we pretrain a massively multilingual document encoder as a hierarchical transformer model (HMDE) in which a shallow document transformer contextualizes sentence representations produced by a state-of-the-art pretrained multilingual sentence encoder. We leverage Wikipedia as a readily available source of comparable documents for creating training data, and train HMDE by means of a cross-lingual contrastive objective, further exploiting the category hierarchy of Wikipedia for creation of difficult negatives. We evaluate the effectiveness of HMDE in two arguably most common and prominent cross-lingual document-level tasks: (1) cross-lingual transfer for topical document classification and (2) cross-lingual document retrieval. HMDE is significantly more effective than (i) aggregations of segment-based representations and (ii) multilingual Longformer. Crucially, owing to its massively multilingual lower transformer, HMDE successfully generalizes to languages unseen in document-level pretraining.

Introduction

Current massively multilingual models, such as XLM-R or mBERT, typically process text as flat sequences of tokens. This approach faces two significant hurdles when applied to documents. First, it ignores the inherent hierarchical structure of documents, where tokens form sentences and sentences form paragraphs. Second, standard transformers are limited by a strict input length (usually 512 tokens), forcing researchers to either truncate text or use sliding window approaches that lose global context.

While sparse-attention models like Longformer address long sequences, they often require task-specific fine-tuning and struggle with unsupervised tasks like document retrieval. To address this, we propose the Hierarchical Multilingual Document Encoder (HMDE), a general-purpose encoder designed for both classification and retrieval across a wide array of languages.

Methodology: The HMDE Architecture

The HMDE architecture is a hierarchical transformer designed to capture document-level semantics by building upon sentence-level representations.

The HMDE architecture utilizes a sentence-level transformer followed by a document-level transformer to produce document embeddings.

The model consists of two primary levels:

  1. Lower Level: We utilize LaBSE (Language-Agnostic BERT Sentence Embedding), a state-of-the-art sentence encoder, to generate representations for individual sentences within a document.
  2. Upper Level: A shallow, 2-layer transformer contextualizes these sentence embeddings. A special [DBOS] (Document Beginning of Sentence) token is introduced to aggregate the final document-level embedding.

Contrastive Pretraining

We train HMDE using an InfoNCE loss on Wikipedia data. Positive pairs consist of the same Wikipedia concept in different languages (e.g., English and Italian pages for "New York"). To enhance the model's discriminative power, we incorporate hard negatives—documents from the same Wikipedia category that represent different concepts. This forces the model to learn fine-grained topical distinctions rather than simple language-based clustering.

Experimental Setup

We pretrain on data from Wiki-40B, creating two variants: XLW-4L (1.1M triples from 4 high-resource languages) and XLW-12L (592K triples from 12 diverse languages). Evaluation is conducted on two benchmarks:

  • MLDOC: A subset of Reuters news stories in 8 languages for topical classification.
  • CLEF-2003: A cross-lingual information retrieval (CLIR) benchmark covering 9 language pairs.

Results and Discussion

HMDE demonstrates superior performance across both classification and retrieval tasks. In document classification, HMDE achieved an average accuracy of 86.8%, surpassing multilingual Longformer (86.1%) and the LaBSE sliding window baseline (84.0%).

The performance gap is most pronounced in the unsupervised retrieval task. As shown in the table below, HMDE significantly outperforms existing baselines.

Cross-lingual document retrieval (CLIR) results (MAP) across various language pairs.

Key Insights

  • Hierarchy vs. Flatness: Explicitly modeling documents as sequences of sentences is more effective than "bag of segments" or flat sparse-attention mechanisms.
  • Zero-Shot Generalization: By initializing with LaBSE (supporting 109 languages), HMDE generalizes to languages not seen during document-level pretraining, such as Finnish.
  • Data Volume over Diversity: Training on a larger corpus of 4 related languages (XLW-4L) proved more effective than a smaller, more diverse set (XLW-12L), suggesting that the volume of comparable data is crucial for document-level logic.
  • Retrieval Excellence: While Longformer is competitive in fine-tuned classification, it fails to produce effective "out of the box" embeddings for retrieval, where HMDE excels with a MAP of 0.249 compared to Longformer's 0.109.

Conclusion

HMDE provides a robust, general-purpose solution for multilingual document encoding. While it remains constrained by the language support of its underlying sentence encoder and the availability of Wikipedia data for certain languages, its hierarchical approach and contrastive training strategy offer a significant step forward for cross-lingual document-level NLP. Future work may explore cross-encoder variants or strategies to expand language coverage beyond the initial pretraining set.

Summary updated

Abstract

Massively multilingual pretrained transformers (MMTs) have tremendously pushed the state of the art on multilingual NLP and cross-lingual transfer of NLP models in particular.While a large body of work leveraged MMTs to mine parallel data and induce bilingual document embeddings, much less effort has been devoted to training general-purpose (massively) multilingual document encoder that can be used for both supervised and unsupervised documentlevel tasks.In this work, we pretrain a massively multilingual document encoder as a hierarchical transformer model (HMDE) in which a shallow document transformer contextualizes sentence representations produced by a stateof-the-art pretrained multilingual sentence encoder.We leverage Wikipedia as a readily available source of comparable documents for creating training data, and train HMDE by means of a cross-lingual contrastive objective, further exploiting the category hierarchy of Wikipedia for creation of difficult negatives.We evaluate the effectiveness of HMDE in two arguably most common and prominent crosslingual document-level tasks: (1) cross-lingual transfer for topical document classification and (2) cross-lingual document retrieval.HMDE is significantly more effective than (i) aggregations of segment-based representations and (ii) multilingual Longformer.Crucially, owing to its massively multilingual lower transformer, HMDE successfully generalizes to languages unseen in document-level pretraining.We publicly release our code and models. 1 .
BibTeX
@misc{litschko2023general,
  author = {Onur Galoğlu Robert Litschko and Robert Litschko and Goran Glavašš},
  title = {{A General-Purpose Multilingual Document Encoder}},
  year = {2023},
  doi = {10.18653/v1/2023.mrl-1.4},
}
Figure