TL;DR
This paper tackles the challenge of translating customer support chats, which are notoriously informal and context-dependent. The authors introduce TOWERCHAT, a fine-tuned LLM that leverages conversational history for better translations, and a novel quality-aware decoding strategy using a context-aware metric (CONTEXTCOMET) to select the best translation. Their approach significantly outperforms existing methods, even outperforming much larger models, and achieves top human evaluation scores.
What Problem Does This Paper Address?
Translating customer support chats presents a unique challenge compared to traditional machine translation tasks like news or Wikipedia. These conversations are often dynamic, informal, and heavily reliant on context. Previous attempts to apply standard MT models to this domain have struggled to effectively utilize this crucial conversational history, leading to issues like lexical inconsistency and incoherence. Large Language Models (LLMs) show promise due to their ability to handle context, but their application to bilingual chat translation, especially in leveraging conversational context for improved accuracy, remains underexplored. This paper aims to bridge this gap by developing methods to adapt LLMs for this specific task and enhance their contextual understanding.
Methods
The authors propose two main strategies:
- Context-Augmented Training (TOWERCHAT): They fine-tune a strong LLM (TOWERBASE 7B) on a specially constructed dataset of chat conversations. This dataset includes "context-augmented instructions," where the model is prompted with preceding turns of the conversation. This process aims to imbue the model with the ability to better understand and utilize conversational context during translation.
- Quality-Aware Decoding (QAD) with Context-Aware Metrics: For inference, they employ a novel approach called Quality-Aware Decoding (QAD). Instead of simply selecting the first translation generated by the model (greedy decoding), QAD generates a pool of candidate translations and then uses a specialized MT evaluation metric to select the best one. Crucially, they use
CONTEXTCOMET, a metric designed to correlate better with human judgment for out-of-English chat translations by incorporating contextual information. This allows the model to choose a translation that is not only fluent but also contextually appropriate.
Datasets
The primary dataset used is the WMT24 Chat MT shared task dataset, which consists of bilingual customer support chats between English-speaking agents and clients speaking Portuguese, French, Italian, Dutch, or Korean. The paper notes that this dataset exhibits characteristics distinct from traditional MT datasets, such as informal language and a higher degree of contextual dependency. They also mention using the standard WMT23 benchmark for evaluating the general translation capabilities of their fine-tuned model.
Key Insights and Ideas
- Context is King (Especially in Chats): The paper strongly emphasizes that for customer support chats, leveraging conversational context is not just beneficial but essential for high-quality translation. Standard LLMs, even powerful ones, struggle without explicit adaptation to this context.
- In-Domain Fine-tuning Pays Off: Fine-tuning a general-purpose LLM on chat-specific data (TOWERCHAT) leads to significant improvements, even when compared to much larger, general-purpose models. This highlights the importance of domain adaptation.
- Decoding Matters as Much as Training: The QAD strategy, particularly when combined with a context-aware metric like
CONTEXTCOMET, provides a substantial boost in translation quality, even for models that have already been fine-tuned. This suggests that advanced inference techniques can unlock further performance gains. - Smaller, Adapted Models Can Beat Larger Generalists: TOWERCHAT, a 7B parameter model, outperforms the much larger TOWER-V2-70B model on the chat translation task, demonstrating that effective adaptation and inference strategies can make smaller models highly competitive.
- Context-Aware Metrics are Crucial for Evaluation: The success of
CONTEXTCOMETin QAD and its correlation with human judgment underscore the need for evaluation metrics that can properly assess context-dependent translation quality.
Key Results
- Human Evaluation Winner: TOWERCHAT, especially with QAD, was the winner of the WMT24 Chat MT shared task based on human evaluation across all language pairs.
- Automatic Metric Performance: The proposed system consistently outperformed baselines and competing systems on various automatic metrics (COMET, CHRF, METRICX, CONTEXT-QE), achieving statistically significant improvements.
- Outperforming Larger Models: TOWERCHAT with QAD outperformed the much larger TOWER-V2-70B model, even when the latter was used with few-shot examples.
- Contextual Gains: TOWERCHAT showed an average improvement of 4 CHRF points for EN→XX when using context compared to a context-agnostic prompt.
- MUDA F1 Scores: QAD with
CONTEXTCOMETconsistently improved MUDA F1 scores, indicating better resolution of lexical ambiguities, which are common in chat conversations. - Minimal Impact on General Capabilities: Fine-tuning TOWERCHAT resulted in only a minor degradation in general translation capabilities on the WMT23 benchmark, validating the approach's viability.
Limitations and Future Work
- Turn-Level vs. Conversation-Level: While TOWERCHAT performs well, the paper notes a visible drop between turn-level and conversation-level scores, suggesting room for improvement in how well the model leverages context across longer conversations.
- Pronoun Accuracy: Qualitative analysis suggests that pronoun accuracy might be lower due to potential paraphrasing, indicating an area for further investigation.
- Future Work: The authors express interest in exploring under what specific circumstances context is most useful and how TOWERCHAT can leverage it even more appropriately. They also suggest further exploration of QAD with context-aware metrics.
Summary updated