CloudSheep System for WMT24 Discourse-Level Literary Translation

Lisa Liu, Ryan Liu, Angela Tsai, Jingbo ShangWMT 2024 · DOI

Introduction

The translation of Chinese webnovels into English presents a unique set of challenges that extend beyond simple word-for-word substitution. Literary genres such as Xianxia (cultivation) and historical fiction rely heavily on discourse-level consistency and the nuanced interpretation of cultural idioms. Standard Machine Translation (MT) engines often fail in these areas, leading to "identity amnesia"—where character names fluctuate—and clunky, literal translations of figurative language. The CloudSheep system, developed for the WMT24 shared task, addresses these flaws through a multi-stage pipeline combining traditional MT with targeted Large Language Model (LLM) interventions.

Methodology: The CloudSheep Pipeline

The CloudSheep system avoids a "black box" approach, instead utilizing a surgical pipeline to maintain consistency and flow.

1. Name and Honorific Dictionary Construction

To combat inconsistent naming, the system first generates a custom dictionary for each text. It uses the Jieba segmentation tool to identify nouns and tasks GPT-3.5 turbo with identifying recurring names and locations from a random 20% sample of the source text. This is supplemented by a pre-compiled list of honorifics (e.g., "Grandmaster") to ensure titles remain standardized throughout the translation.

2. Idiom Integration and Base Translation

The system identifies Chinese idioms (chengyu) using the CC-CEDICT open-source dictionary. These idioms are replaced with their raw definitions and flagged for later refinement. The remaining text is then translated using a base engine like Google Translate or DeepL to provide a grammatical foundation.

The CloudSheep workflow for integrating name dictionaries and idiom definitions into the translation process.

3. Contextual Rephrasing with GPT-4

The final stage involves using GPT-4 to "smooth" the sentences containing flagged idiom definitions. By acting as a contextual editor, the LLM rephrases the literal dictionary definitions into natural English that fits the tone and flow of the surrounding passage. This targeted use of expensive LLMs (affecting only 2%–11% of the text) optimizes both quality and cost.

Evaluation and Results

The researchers evaluated CloudSheep using the GuoFeng Webnovel Corpus and additional excerpts from popular webnovel platforms.

Automated Metrics and Genre Sensitivity

Automated metrics like BLEU, chrF, and COMET showed that the CloudSheep pipeline performs similarly to standalone engines on general text. However, the system showed particular strength in challenging genres like Martial Arts. In these cases, while standard metrics drop significantly due to the complexity of the source material, the pipeline maintains competitive performance, particularly in semantic similarity metrics like COMET compared to standalone DeepL.

Translation performance metrics in a challenging literary genre, highlighting the pipeline's relative strength.

Human A/B Testing

In contrast to automated scores, human evaluation strongly favored the CloudSheep approach. In A/B testing, the "Pipeline + Google Translate" configuration was most frequently ranked as the top choice. Conversely, standalone DeepL—despite its popularity in fan-translation circles—was often ranked lowest due to its failure to maintain character name consistency.

Conclusion and Future Directions

The CloudSheep system demonstrates that for literary translation, consistency is often more important than literal perfection. By prioritizing discourse-level identity and figurative accuracy, the system provides a more immersive experience for readers. Future work will focus on improving the robustness of the name-sampling process and reducing the inherent randomness of LLM-based rephrasing to ensure deterministic, high-quality outputs across all genres.

Summary updated

BibTeX
@inproceedings{liu2024cloudsheep,
  author = {Lisa Liu and Ryan Liu and Angela Tsai and Jingbo Shang},
  title = {{CloudSheep System for WMT24 Discourse-Level Literary Translation}},
  booktitle = {WMT},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.95},
}
Figure