ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback

Shiyue Zhang, Benjamin Frey, Mohit Bansal

Abstract

We introduce ChrEnTranslate, an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability, two user feedback interfaces for experts and common users respectively, example inputs to collect human translations for monolingual data, word alignment visualization, and relevant terms from the Cherokee-English dictionary. The quantitative evaluation demonstrates that our backbone translation models achieve state-of-the-art translation performance and our quality estimation well correlates with both BLEU and human judgment. By analyzing 216 pieces of expert feedback, we find that NMT is preferable because it copies less than SMT, and, in general, current models can translate fragments of the source sentence but make major mistakes. When we add these 216 expert-corrected parallel texts back into the training set and retrain models, equal or slightly better performance is observed, which indicates the potential of human-in-the-loop learning.

Introduction

While machine translation (MT) has become a ubiquitous tool for major world languages, it remains a significant challenge for "low-resource" languages like Cherokee. Identified as "endangered" by UNESCO, Cherokee faces a vanishing digital footprint. Standard MT models require millions of parallel sentences, but Cherokee datasets are tiny—this study utilized a compiled set of just 17,000 sentences, much of which originates from religious texts like the Cherokee New Testament.

In such low-resource settings, models often "hallucinate" or produce nonsensical results. For language learners or tribal elders, an incorrect translation can be more harmful than none at all. ChrEnTranslate addresses this by integrating automated Quality Estimation (QE) to warn users of unreliable outputs and a corrective feedback loop to allow experts to improve the system.

The ChrEnTranslate System

The researchers developed a dual-engine architecture to balance robustness and fluency:

  1. Statistical Machine Translation (SMT): Built using the Moses framework, SMT is often more robust for "out-of-domain" text that the model hasn't encountered during training.
  2. Neural Machine Translation (NMT): An ensemble of three LSTM-based models with attention mechanisms. Notably, the team found that standard Transformer architectures performed worse than LSTMs on this specific small dataset, highlighting that state-of-the-art methods for high-resource languages do not always translate to low-resource contexts.

To ensure reliability, the system provides a 5-star Quality Estimation (QE) rating. This is achieved through a supervised XGBoost regressor that predicts BLEU scores and an unsupervised method based on the model’s internal confidence.

The ChrEnTranslate interface showing Cherokee-to-English translation with a 3-star quality estimation.

Human-in-the-Loop Feedback

A core innovation of ChrEnTranslate is its dual-interface feedback system. "Common Users" can provide simple helpfulness ratings, while "Experts" can provide full sentence corrections and open-ended comments.

The expert feedback interface allows proficient speakers to provide corrected translations and comments to improve the model.

Analysis of 216 expert corrections revealed that while SMT models tend to excessively copy source words when confused, NMT models are more fluent but prone to major semantic errors. Furthermore, the "Bible bias" in the training data often resulted in archaic English (e.g., "thou," "speaketh"), which the researchers addressed through modernizing post-processing.

Key Results and Impact

The system achieved state-of-the-art performance, with the NMT ensemble reaching a 19.9 BLEU score for Cherokee-to-English translation. Experts overwhelmingly preferred NMT outputs because the models attempted actual translation rather than mere copying.

Crucially, the "human-in-the-loop" approach showed immediate promise. Adding just 216 expert-corrected sentences back into the training set led to measurable improvements, suggesting that for endangered languages, the quality of expert data is far more valuable than sheer quantity. The QE system also proved effective, correlating well with human judgment ($\gamma \ge 0.44$), serving as a vital safety net against misinformation.

Conclusion and Future Work

ChrEnTranslate represents a significant step toward revitalizing the Cherokee language's digital presence. While the data bottleneck remains a primary limitation—models still struggle with complex grammar and diverse topics—the success of the feedback loop points toward a sustainable path for improvement. Future work will explore more sophisticated active learning methods to identify the most helpful sentences for expert translation and aim to expand this framework to other endangered languages globally.

The online demo is available at https://chren.cs.unc.edu/.

Summary updated

Abstract

We introduce ChrEnTranslate, an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability, two user feedback interfaces for experts and common users respectively, example inputs to collect human translations for monolingual data, word alignment visualization, and relevant terms from the Cherokee-English dictionary. The quantitative evaluation demonstrates that our backbone translation models achieve state-of-the-art translation performance and our quality estimation well correlates with both BLEU and human judgment. By analyzing 216 pieces of expert feedback, we find that NMT is preferable because it copies less than SMT, and, in general, current models can translate fragments of the source sentence but make major mistakes. When we add these 216 expert-corrected parallel texts back into the training set and retrain models, equal or slightly better performance is observed, which indicates the potential of human-in-the-loop learning. Our online demo is at https://chren.cs.unc.edu/ , our code is open-sourced at https://github.com/ZhangShiyue/ChrEnTranslate , and our data is available at https://github.com/ZhangShiyue/ChrEn
BibTeX
@inproceedings{zhang2021chrentranslate,
  author = {Shiyue Zhang and Benjamin Frey and Mohit Bansal},
  title = {{ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-demo.33},
}
Figure