Final Submission of SJTULoveFiction to Literary Task

Haoxiang Sun, Tianxiang Hu, Ruize Gao, Jialong Tang, Pei Zhang, Baosong Yang, Rui WangWMT 2024 · DOI

Abstract

Literary translation, particularly for long-form web novels, presents unique challenges in maintaining discourse-level coherence and terminology consistency. The SJTULoveFiction team addressed these issues for the WMT24 Discourse-Level Literary Translation task using a novel multi-agent pipeline. Their approach combines a chunk-based processing strategy with a multi-model ensemble, refined by specialized Translation Editor and Term Proofreader agents. Experimental results demonstrate that this collaborative agent framework significantly outperforms individual state-of-the-art models in both linguistic quality and narrative consistency.

Introduction

Traditional Machine Translation (MT) systems often struggle with the nuances of literature. Unlike technical or news text, literary content relies heavily on discourse-level coherence, where sentences must flow naturally over long spans. Furthermore, maintaining terminology consistency—ensuring a character's name or a specific world-building element remains identical throughout a chapter—is critical for reader immersion. Standard sentence-level MT often loses this "big picture" context. This paper details a system designed to leverage the massive context windows and reasoning capabilities of Large Language Models (LLMs) to preserve the narrative thread in long-form translations.

Methodology

The SJTULoveFiction system utilizes a sophisticated four-step pipeline designed to move beyond simple sequence-to-sequence translation.

1. Chunk-Based Strategy

To provide models with sufficient context, the team implemented a chunking strategy. Instead of translating individual sentences, text is grouped into segments ranging from 5 to 80 sentences. This allows the LLMs to understand surrounding stylistic and grammatical cues, leading to more cohesive output.

2. Multi-Model Translation and SFT

The system employs an ensemble of three top-tier models: a fine-tuned Qwen2-72B (trained on the specialized GuoFeng Webnovel Corpus), Claude-3.5, and GPT-4o. This diversity ensures a broad range of stylistic candidates for the subsequent refinement stages.

3. Translation Editor Agent

Recognizing that different models possess varying strengths, the team deployed GPT-4o as a "Translation Editor." This agent evaluates the source text alongside the three candidate translations, merging the most accurate and stylistically appropriate segments into a single, optimized version.

The multi-agent pipeline architecture, showing the flow from source text through chunking, multi-model translation, and agent-led refinement.

4. Terminology Intervention

To anchor the translation, a Term Proofreader Agent was developed. This agent extracts key terms from the merged translation, facilitates the selection of an "optimal" translation for each term, and then performs a global sweep to enforce these terms consistently across the entire chapter.

Key Insights and Results

The "Goldilocks" Chunk Size

Research into context windows revealed that more context is not always better. The team identified a "sweet spot" for chunk sizes; while training on 10-sentence chunks was effective for the fine-tuned Qwen model, inference on 5-sentence chunks often provided the best balance of context and accuracy for general-purpose models like Claude and GPT-4o.

Superiority of Merging

The translation merging strategy proved highly effective. In Linguistic Quality Ratings (LQR), the merged output achieved a "Completely correct and fluent" (LQR5) score 24% of the time, a significant improvement over the 9% achieved by GPT-4o acting alone.

Consistency Gains

The impact of the Term Proofreader Agent was particularly striking. Before intervention, a single Chinese term often resulted in an average of 3.73 different English variations throughout a text. Following the agent's intervention, this variance dropped to 1.12, dramatically improving the narrative's internal consistency.

The Term Proofreader process: extracting terms from the merged translation, selecting the optimal version, and applying it globally to ensure consistency.

Conclusion and Future Work

The SJTULoveFiction system demonstrates that the future of literary MT lies in refinement rather than just generation. By treating LLMs as evaluators and editors within a multi-agent framework, the system successfully synthesizes the creative strengths of multiple models while enforcing strict terminology constraints. Future work will focus on optimizing techniques for ultra-long context spans and automating the "optimal" term selection process to further reduce the need for human-in-the-loop intervention.

Summary updated

BibTeX
@inproceedings{sun2024final,
  author = {Haoxiang Sun and Tianxiang Hu and Ruize Gao and Jialong Tang and Pei Zhang and Baosong Yang and Rui Wang},
  title = {{Final Submission of SJTULoveFiction to Literary Task}},
  booktitle = {WMT},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.96},
}
Figure