TL;DR
This paper proposes a new approach to evaluating machine translation (MT) and automatic summarization (AS) by moving beyond sentence-level metrics to consider document-level discourse. The authors argue that traditional metrics fail to capture document quality and introduce discourse-based features that show promise for more accurate evaluation.
What Problem Does This Paper Address?
Evaluating the quality of machine-generated text, like translations and summaries, is crucial but challenging. Current automatic evaluation metrics, such as BLEU and ROUGE, rely on comparing system output to human-written references. While useful, these metrics have limitations: they can penalize good translations that differ significantly from references and are impractical for scenarios where no reference is available (e.g., real-time translation or summarizing web pages).
Quality Estimation (QE) offers an alternative by using machine learning to predict quality without references. However, most existing QE research focuses on the sentence level. This paper argues that this sentence-level focus misses important document-level information, particularly discourse, which is how sentences connect and form a coherent whole. Machine translation systems often process text sentence by sentence, potentially losing this discourse structure. Therefore, evaluating at the document level, incorporating discourse, is essential for a more comprehensive quality assessment.
Methods
The paper's core contribution lies in exploring document-level Quality Estimation (QE) by leveraging discourse phenomena.
- Document-Level Quality Labels: The authors question the suitability of traditional metrics (like BLEU, TER, METEOR, and averaged HTER) as document-level quality labels. They hypothesize that simply averaging sentence-level scores doesn't adequately reflect document quality, as some sentences might be more critical than others. They propose that a more sensitive quality label is needed, potentially incorporating summarization or information retrieval techniques to weight important sentences.
- Discourse Features: The paper investigates using discourse phenomena as features for document-level QE. These phenomena include:
- Lexical Cohesion: Analyzing word repetitions, synonyms, and collocations to understand how sentences are linked.
- Latent Semantic Analysis (LSA): Using LSA to capture semantic relationships within documents, which can indirectly reflect cohesion.
- Rhetorical Structure Theory (RST): Exploring RST to model the hierarchical structure of discourse and how different parts of a text relate to each other.
- Entity-Grid Models: (Mentioned as a potential area, though not deeply explored in this paper).
- Coreference Resolution: Identifying how pronouns and other referring expressions link entities across sentences.
- Discourse Connectives: Analyzing words like "however," "therefore," and "because" that signal relationships between clauses and sentences.
- Machine Learning Models: The QE models are trained using supervised machine learning, typically regression algorithms, where the extracted features are used to predict quality scores.
Datasets
The paper mentions using the following datasets for evaluation:
- WMT2013 QE shared task corpus: English-Spanish translations.
- LIG corpus: French-English translations with post-editions.
The authors note that while some discourse resources (like RST parsers) are currently only available for English, their research aims to explore their impact on document-level QE, considering English as either the source or target language.
Key Insights and Ideas
- Sentence-level metrics are insufficient for document quality: Averaging sentence-level scores doesn't capture the nuanced quality of an entire document, as critical sentences might be poorly translated without significantly impacting the average score.
- Discourse phenomena are strong indicators of document quality: Features derived from discourse analysis (like lexical cohesion, RST, and connectives) show higher correlation with document-level quality scores (HTER) than traditional sentence-level features or even some successful features from existing QE frameworks (QuEst).
- Discourse models can be language-independent: Latent variable models like LDA and LSA are fast, language-independent, and can be used to model document coherence, making them valuable for QE.
- The potential for broader NLP applications: The discourse models developed for MT QE could also be beneficial for evaluating other NLP tasks like Automatic Summarization (AS) and Readability Assessment (RA).
Key Results
The paper presents results from an experiment on the LIG corpus, showing the correlation of various discourse features with document-level HTER.
- Discourse features outperform traditional ones: Features considering discourse phenomena counts performed better than QuEst features in scenarios with 10 and 20 documents.
- Lexical Cohesion and LSA show promise: In other scenarios, Lexical Cohesion (LC) and LSA features were the best performers.
- Traditional metrics show low variation: The study found that traditional MT evaluation metrics exhibited low standard deviation across documents, suggesting they are not sensitive enough to differentiate document quality.
Limitations and Future Work
- Limited availability of discourse resources: Some discourse analysis tools, particularly for RST and Entity-Grid models, are currently only available for English, limiting their immediate application to other languages.
- Ongoing research: The study on discourse features and their correlation with HTER is described as "on-going," implying further analysis and refinement are planned.
- Future Directions:
- Developing more sensitive document-level quality labels.
- Exploring the applicability of discourse models for QE across different languages.
- Investigating how discourse models can be used to improve the evaluation of Automatic Summarization (AS) and Readability Assessment (RA).
- Integrating document-level and sentence-level prediction and features within existing QE frameworks like QuEst.
Summary updated