Findings of the WMT 2018 Biomedical Translation Shared Task: Evaluation on Medline test sets

Mariana Neves, Antonio Jimeno Yepes, Aurélie Névéol, Cristian Grozea, Amy Siu, Madeleine Kittner, Karin VerspoorEMNLP 2018

TL;DR

The WMT 2018 Biomedical Translation Shared Task focused on evaluating machine translation systems for biomedical texts using Medline and EDP test sets across six language pairs. While overall performance saw some improvements, particularly with Transformer models and domain-specific resources, manual evaluations revealed that automatic translations sometimes surpassed human references, highlighting the growing sophistication of MT in specialized domains.

What Problem Does This Paper Address?

Access to scientific literature is often limited by language barriers. Machine translation (MT) offers a solution, but its effectiveness in highly specialized domains like biomedicine, with its unique terminology and complex sentence structures, remains a challenge. This paper addresses the need for robust evaluation of MT systems specifically for biomedical texts, aiming to understand the current state-of-the-art, identify areas for improvement, and foster progress in this critical field. The challenge specifically targets the translation of scientific abstracts, which are crucial for disseminating research findings.

Methods

The WMT 2018 Biomedical Translation Shared Task employed a multi-faceted approach to evaluate MT systems.

  • Test Set Creation: New test sets were constructed from Medline and EDP scientific publications. For Medline, abstracts were extracted and automatically aligned using tools like Stanford CoreNLP and GMA, with manual checking and refinement for quality. The EDP test set was derived from previous work, with manually segmented sentences for improved accuracy.
  • System Submissions: Six teams participated, submitting a total of 39 runs. Systems utilized various architectures, including Transformer-based neural machine translation (NMT) and statistical machine translation (SMT) with Moses. Many teams leveraged biomedical and news corpora for training, with some employing transfer learning and ensemble methods.
  • Automatic Evaluation: BLEU scores were the primary metric for automatic evaluation, calculated at the sentence level. This was performed on both the full set of automatically aligned sentences and a subset of perfectly aligned sentences to assess the impact of alignment quality.
  • Manual Evaluation: A crucial component involved manual evaluation by native speakers. This included assessing the quality of automatic sentence alignments and performing pairwise comparisons of translations (system vs. reference, system vs. system) to determine relative quality. A 3-way ranking task was used for this purpose.

Datasets

  • Medline: A primary source for test sets, comprising scientific abstracts. These were processed to create parallel corpora for six language pairs: English with Chinese, French, German, Portuguese, Romanian, and Spanish.
  • EDP: Another source for test sets, also containing scientific publications.
  • Training Data: Participating teams utilized various in-domain and out-of-domain corpora, including scientific articles from Scielo, the UFAL medical corpus, the EMEA corpus, and Brazilian theses and dissertations.

Key Insights and Ideas

  • Transformer Dominance: Transformer-based NMT architectures continued to show strong performance, especially when combined with domain-specific training data and resources.
  • Domain-Specific Resources Matter: Teams that effectively utilized biomedical corpora and terminological resources generally achieved higher scores, underscoring the importance of specialized data for high-quality biomedical translation.
  • Manual vs. Automatic Alignment Impact: Evaluating translations based only on perfectly aligned sentences led to significant improvements in BLEU scores compared to using all automatically aligned sentences. This highlights the limitations of automatic alignment quality in creating reliable test sets.
  • Manual Evaluation Surprises: In some cases, manual evaluations indicated that automatic translations were rated as equal to or even better than the human-generated reference translations, particularly for German and Spanish. This suggests that MT systems are becoming highly competitive in certain biomedical translation scenarios.
  • Language Pair Performance Varies: Performance was strongly correlated with the availability of parallel data. Language pairs with more abundant biomedical parallel corpora (e.g., en/es, pt/en) generally saw higher BLEU scores.
  • Diacritics Normalization for Romanian: For Romanian, normalizing diacritics in both the test set and submissions was crucial for accurate BLEU scoring, as minor diacritic variations can lead to significant score degradation.

Key Results

  • BLEU Scores: While specific scores varied by language pair and team, some BLEU scores were higher than in the previous year, particularly for systems leveraging Transformer models and biomedical resources.
  • Manual Evaluation Findings:
    • For German, 80% of automatic translations were rated as equally good or better than the reference.
  • For es/en and pt/en, manual evaluations sometimes diverged from automatic rankings, with human evaluators favoring systems that achieved lower BLEU scores but were deemed better quality.
  • In several instances, automatic translations were ranked higher than the reference translations by human evaluators.
  • Test Set Quality: The manual validation of sentence alignments revealed issues, particularly with Spanish and German test sets, where alignment quality was lower (around 70% and 65% correctly aligned, respectively).

Limitations and Future Work

  • Data Availability: Limitations in the quantity of available parallel abstracts for some languages were noted. The paper suggests that future test sets could benefit from manual translation efforts or the inclusion of new parallel corpora.
  • Test Set Format: The BioC format used for data exchange presented difficulties for sentence alignment, prompting consideration of alternative formats for future challenges.
  • Overlap with Training Data: The possibility of test set sentences appearing in training data was raised as a potential factor influencing results, particularly for the TGF TALP UPC team.
  • Future Work: The authors suggest exploring new parallel corpora, potentially including full texts from sources like Scielo, and considering manual translation for future test set creation. Investigating alternative data exchange formats is also planned. The increasing quality of automatic translations suggests continued focus on nuanced word choices and semantic accuracy.

Summary updated

BibTeX
@inproceedings{neves2018findings,
  author = {Mariana Neves and Antonio Jimeno Yepes and Aurélie Névéol and Cristian Grozea and Amy Siu and Madeleine Kittner and Karin Verspoor},
  title = {{Findings of the WMT 2018 Biomedical Translation Shared Task: Evaluation on Medline test sets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure