DOLFIN - Document-Level Financial Test-Set for Machine Translation

Mariam Nakhlé, Marco Dinarelli, Raheel Qader, Emmanuelle Esperança-Rodier, Hervé BlanchonNAACL 2025 · DOI

Abstract

Machine Translation (MT) evaluation has traditionally relied on sentence-level benchmarks, which often fail to capture the complexities of specialized domains like finance. This paper introduces DOLFIN, a novel document-level test set designed to evaluate MT models on aligned sections rather than isolated sentences. Spanning five European language pairs, DOLFIN enables a more realistic assessment of context-aware translation, terminology consistency, and information reorganization.

Introduction

While general-purpose MT has seen significant advancements, specialized fields like finance present unique challenges. In financial reports, terms like the French couverture can mean "blanket" in general usage but "hedge" in a financial context. Sentence-level evaluation creates a blind spot, as models lack the broader context necessary to resolve such polysemy. Furthermore, existing document-level datasets often rely on "perfect sentence alignment," which does not reflect the fluid nature of human translation where sentences are frequently merged or split. DOLFIN addresses these gaps by providing a benchmark based on realistic section-level alignments.

Methods

The development of DOLFIN involved a multi-stage pipeline designed to process complex financial documents:

  1. Extraction: Text was extracted from financial PDFs using Apryse, preserving structural elements like titles and tables.
  2. Section-Level Alignment: Instead of strict sentence-to-sentence mapping, entire sections were aligned using LASER embeddings and cosine similarity, reflecting the "imperfect" but realistic nature of professional translation.
  3. Filtering & Quality Control: Rigorous filtering removed repetitive legal boilerplate (approximately 80% of the initial data). Quality was further ensured using the Comet-kiwi quality estimation model.
  4. Linguistic Tagging: Segments were tagged for specific challenges, such as anaphora, ellipsis, and polysemy, using a combination of automated tools and Large Language Models (LLMs).

The DOLFIN Dataset

The resulting dataset covers five language pairs: En-Fr, En-De, Fr-Es, En-Es, and En-It. It includes 15 different document types, such as Annual Reports and Key Information Documents (PRIIPs).

Table 1: Statistics of the DOLFIN dataset across language pairs.

As shown in Table 1, the dataset maintains a substantial volume of aligned sections, with an average of 1,951 segments per language pair and a high average sentence count per segment, providing ample context for evaluation.

Experimental Results and Insights

The researchers evaluated several state-of-the-art models in two configurations: "Per sent" (sentence-by-sentence) and "Full seg" (entire section at once).

Table 2: Performance comparison (Comet-slide scores) between sentence-level and section-level translation.

Contextual Gains and the "Context Penalty"

Large models like GPT-4o and Llama 3-70b showed significant performance gains when provided with full section context. For instance, Llama 3-70b improved by +5.57 points on the Fr-Es pair. Qualitatively, these models were better at resolving pronoun genders and maintaining consistency.

However, a "context penalty" was observed for smaller models. Llama 3.1-8b often performed worse in the "Full seg" setting, sometimes producing incoherent output. This suggests a minimum capacity threshold is required for models to effectively leverage document-level context.

Domain Expertise vs. General Intelligence

Despite improvements, models still struggle with domain-specific terminology. For example, Llama 3-70b incorrectly translated the financial term "Charges" (fees) as "criminal charges" in German. Additionally, formatting remains a challenge; models frequently break Markdown table structures when processing long segments.

Conclusion

DOLFIN provides a critical resource for advancing document-level MT in the financial sector. By moving beyond sentence-level constraints, it highlights the strengths and current limitations of LLMs in handling complex, context-dependent data. Future work will focus on expanding the dataset with manual annotations and conducting human evaluations to further refine these benchmarks.

Summary updated

BibTeX
@inproceedings{nakhle2025dolfin,
  author = {Mariam Nakhlé and Marco Dinarelli and Raheel Qader and Emmanuelle Esperança-Rodier and Hervé Blanchon},
  title = {{DOLFIN - Document-Level Financial Test-Set for Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.307},
}
Figure