Front Matter

Barry Haddow, Tom Kocmi, Philipp Koehn, Christof MonzOpenalex Unknown 2024 · DOI · OpenAlex

Abstract

The Ninth Conference on Machine Translation (WMT 2024), held in Lisbon, Portugal, serves as a premier venue for machine translation (MT) research. This year’s proceedings encompass 25 full research papers and 96 shared task presentations, selected from a competitive pool of 54 full research submissions. Central to the conference were 12 shared tasks covering general translation, low-resource languages (including specific tracks for Spain and India), chat, biomedical, and multi-modal translation, alongside critical evaluation tasks in metrics and quality estimation. These proceedings summarize the results of these tasks, the datasets utilized, and the methodologies employed by participating teams, providing a comprehensive overview of the current state of the art in MT.

Introduction: The LLM Era in Machine Translation

The WMT 2024 proceedings mark a definitive transition in the field: the "LLM Era" has officially arrived. While specialized Neural Machine Translation (NMT) architectures have long dominated the landscape, Large Language Models (LLMs) are now closing the quality gap, particularly in quality estimation and low-resource translation. However, the conference highlights that MT is far from "solved." Researchers are currently grappling with a "crisis of evaluation" as traditional metrics struggle to keep pace with the fluency of LLM outputs, and unique challenges persist in document-level and multi-modal contexts.

The Shared Tasks: Benchmarking the Frontier

WMT is defined by its shared tasks, which provide the industry standard for benchmarking. In 2024, the conference organized 12 tasks across three primary domains:

  1. Translation Tasks: These included General Translation, Chat, and Biomedical tracks, as well as specialized tracks for low-resource languages of Spain and the MultiIndic22MT task for 22 Indian languages.
  2. Evaluation Tasks: Focused on Metrics, Quality Estimation (QE), and MT Test Suites, these tasks investigated whether automated "judges" remain reliable in the face of increasingly human-like machine output.
  3. Data Initiatives: The "Open Language Data Initiative" focused on creating high-quality seed datasets for under-represented languages like Karakalpak, Erzya, and Wu Chinese.

Methodological Evolution

The research presented at WMT 2024 showcases a shift toward integrating LLM capabilities into the MT pipeline. Key methodologies highlighted by participating teams include:

  • LLM Finetuning: Extensive use of QLoRA (Quantized Low-Rank Adaptation) and CPO (Contrastive Preference Optimization) to refine general-purpose models for translation tasks.
  • Model Merging: Techniques to combine the strengths of multiple specialized models to improve overall robustness.
  • Minimum Bayes Risk (MBR) Decoding: A strategy increasingly used to select the most statistically sound translation by comparing multiple candidate outputs, often combining traditional NMT and modern LLM strengths.

The Evaluation Paradox and Metrics under Pressure

A recurring theme throughout the conference was the "LLM Paradox." While LLMs excel at linguistic fluency, they remain prone to domain-specific terminology errors and "hallucinations" that traditional NMT systems often avoid. This has led to a breakdown in traditional metrics like BLEU. As models become more human-like, the statistical differences that these metrics rely on are disappearing. Consequently, the field is moving toward "Quality Estimation" (QE), where models are trained to evaluate the output of other models, though this introduces new concerns regarding metric bias.

Low-Resource Resilience and Multi-Modal Context

Despite the power of LLMs, low-resource translation continues to require sophisticated data engineering. The proceedings demonstrate that "Hybrid Distillation"—combining Rule-Based MT with NMT—remains a potent strategy for languages with minimal digital footprints. Furthermore, the inclusion of tasks like "English-to-Lowres Multi-Modal Translation" and "Discourse-Level Literary Translation" indicates a shift toward understanding broader context, whether that context is visual information or the narrative flow of a long-form text.

Key Results and Future Challenges

The findings indicate that LLM-based systems are now top-tier competitors in almost every track, especially in General Translation and Quality Estimation. However, several critical frontiers remain:

  • Metric Bias: There is an urgent need to mitigate bias in automated metrics, particularly when used in MBR decoding.
  • Security: As MT systems adopt LLM architectures, they become vulnerable to "Prompt Injection Attacks," a new security concern for the community.
  • The Understanding Gap: For low-resource languages, scaling models is not a panacea. Research suggests that LLMs still struggle with deep retrieval and linguistic understanding in these contexts, emphasizing the need for diverse training data through initiatives like FLORES+ and the Open Language Data Initiative.

In summary, WMT 2024 illustrates a field in flux—embracing the generative power of LLMs while simultaneously building the rigorous evaluation frameworks and data foundations necessary to ensure translation remains accurate, secure, and inclusive.

Summary updated

Abstract

in Lisbon, Portugal.The focus of our conference is to bring together researchers from the area of machine translation and invite selected research papers to be presented at the conference.Prior to the conference, in addition to soliciting relevant papers for review and possible presentation, we conducted 12 shared tasks.These consisted of 8 translation tasks: General Translation, Translation into Low-Resource Languages of Spain, Low-Resource Indic Language Translation, Chat Translation, Biomedical Translation, MultiIndic22MT, Non-Repetitive Translation, English-to-Lowres Multi-Modal Translation, three evaluation tasks: Metrics, MT Test Suites, Quality Estimation, and finally the Open Language Data Initiative.The results of all shared tasks were announced at the conference, and these proceedings also include overview papers for the shared tasks, summarizing the results, as well as providing information about the data used and any procedures that were followed in conducting or scoring the tasks.In addition, there are short papers from each participating team that describe their underlying system in greater detail.Like in previous years, we have received a far larger number of submissions than we could accept for presentation.WMT 2024 has received 54 full research paper submissions (not counting withdrawn submissions).In total, WMT 2024 featured 25 full research paper presentations and 96 shared task presentations.
BibTeX
@inproceedings{haddow2024front,
  author = {Barry Haddow and Tom Kocmi and Philipp Koehn and Christof Monz},
  title = {{Front Matter}},
  booktitle = {Openalex Unknown},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.0},
}
Figure