Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popovic, Mariya Shmatova, Steinthór Steingrímsson, Vilém Zouhar

Abstract

This overview paper presents the results of the General Machine Translation Task organised as part of the 2024 Conference on Machine Translation (WMT). In the general MT task, participants were asked to build machine translation systems for any of 11 language pairs, to be evaluated on test sets consisting of three to five different domains. In addition to participating systems, we collected translations from 8 different large language models (LLMs) and 4 online translation providers. We evaluate system outputs with professional human annotators using a new protocol called Error Span Annotations (ESA).
BibTeX
@inproceedings{kocmi2024findings,
  author = {Tom Kocmi and Eleftherios Avramidis and Rachel Bawden and Ondrej Bojar and Anton Dvorkovich and Christian Federmann and Mark Fishel and Markus Freitag and Thamme Gowda and Roman Grundkiewicz and Barry Haddow and Marzena Karpinska and Philipp Koehn and Benjamin Marie and Christof Monz and Kenton Murray and Masaaki Nagata and Martin Popel and Maja Popovic and Mariya Shmatova and Steinthór Steingrímsson and Vilém Zouhar},
  title = {{Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet}},
  booktitle = {WMT},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.1},
}

Loading related content...

Figure