Robust Fragment-Based Framework for Cross-lingual Sentence Retrieval

Nattapol Trijakwanich, Peerat Limkonchotiwat, Raheem Sarwar, Wannaphong Phatthiyaphaibun, Ekapol Chuangsuwanich, Sarana NutanongEMNLP 2021 · DOI

Abstract

Cross-lingual Sentence Retrieval (CLSR) is a critical task for building parallel corpora, yet standard multilingual sentence encoders often suffer from significant performance degradation when applied to out-of-domain (OOD) data. This paper introduces the Robust Fragment-level Representation (RFR) framework, which shifts the matching granularity from entire sentences to sub-sentence fragments ($n$-grams). By integrating fragment-level details with global sentence context and utilizing a probabilistic search mechanism with entropy filtering, RFR achieves substantial performance gains. Experimental results across multiple OOD datasets show that RFR improves base encoders in over 85% of cases, demonstrating remarkable robustness in challenging retrieval scenarios.

Introduction

Cross-lingual Sentence Retrieval (CLSR) involves identifying translation pairs across different languages within large, comparable corpora. This process is foundational for developing high-quality machine translation models and other multilingual NLP tools. Currently, state-of-the-art methods rely on multilingual sentence encoders like LASER or LaBSE to map sentences into a shared embedding space. While these models perform well on data similar to their training sets, their performance often craters when faced with Out-of-Domain (OOD) data. In such cases, global similarity scores frequently fail to distinguish between true translations and sentences that are merely topically similar, leading to a high rate of false positives.

The RFR Framework

The authors propose the Robust Fragment-level Representation (RFR) framework to move beyond the limitations of single-vector sentence representations. Instead of comparing one vector per sentence, RFR treats a sentence as a collection of overlapping fragments.

The RFR framework architecture, illustrating the transition from sentence-level to fragment-level representation and the subsequent probabilistic aggregation.

The framework follows a multi-stage pipeline:

  1. Fragmentation and Encoding: Sentences are broken into $n$-gram fragments (with $n=6$ identified as the optimal size) using a sliding window. Each fragment is encoded using a base multilingual encoder.
  2. Hybrid Representation: To prevent "accidental matching" of common phrases, the global sentence embedding is concatenated with each fragment embedding. This ensures the fragment retains its original context.
  3. Probabilistic k-Nearest Neighbor (PkNN) Search: Instead of simple distance metrics, the framework calculates the probability that a query fragment belongs to a specific target sentence, treating sentence IDs as distinct classes.
  4. Aggregation and Entropy Filtering: Probabilities are aggregated across all fragments in a sentence. An Entropy Filter is employed to discard uninformative fragments that match too many different sentences, focusing the final score on the most distinctive parts of the text.

Experimental Setup

The RFR framework was evaluated across four language pairs (French, German, Arabic, and Thai to English) using three distinct OOD datasets: JW300 (religious-society magazines), QED (educational video subtitles), and TED2020 (TED talk transcripts). Additionally, the framework's flexibility was tested on a Cross-lingual Question Answering (CLQA) task using the Xquad dataset to evaluate document and paragraph-level retrieval.

Results and Discussion

The RFR framework consistently outperformed base encoders (m-USE, LASER, and LaBSE) across nearly all experimental conditions, improving performance in 32 out of 36 comparisons.

Comprehensive performance comparison (F1 scores) of base encoders versus RFR-enhanced versions across multiple datasets and languages.

As shown in the results table, the gains are particularly massive for low-resource languages and highly divergent domains. For example, on the JW300 Thai-English dataset, the base m-USE model achieved a dismal F1 score of 8.5, which jumped to 63.5 when integrated with the RFR framework.

Key insights from the evaluation include:

  • Granularity Matters: Sentence-level embeddings are often too "blurry" for OOD data. Fragment-level analysis allows the model to find specific anchor phrases that confirm a translation.
  • Contextual Robustness: Appending the global sentence embedding provides a necessary "sanity check," ensuring that sub-sentence matches are only considered if the overall sentences are semantically related.
  • Precision and Filtering: RFR is exceptionally good at increasing precision. Its filtering mechanisms (entropy and pairing thresholds) make it much more resilient to "non-pairing" distractor sentences compared to raw encoder scores.

Conclusion and Future Work

The RFR framework significantly enhances the robustness of cross-lingual sentence retrieval in out-of-domain scenarios. While the approach introduces a computational trade-off—retrieval speed is approximately 30x to 70x slower than base models due to the increased number of search operations—the improvements in F1 scores are substantial. Future research may explore sub-sentence matching to handle partial translations and methods to optimize the computational efficiency of the fragment-based search process.

Summary updated

BibTeX
@inproceedings{trijakwanich2021robust,
  author = {Nattapol Trijakwanich and Peerat Limkonchotiwat and Raheem Sarwar and Wannaphong Phatthiyaphaibun and Ekapol Chuangsuwanich and Sarana Nutanong},
  title = {{Robust Fragment-Based Framework for Cross-lingual Sentence Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.80},
}
Figure