TL;DR
This paper tackles the challenge of evaluating the subjective quality of machine-generated text (MGT), specifically machine translation (MT). By conducting a crowdsourced study and employing exploratory factor analysis, the authors identify four key perceptual dimensions: precision, complexity, grammaticality, and transparency, which are crucial for understanding user experience with MGT.
What Problem Does This Paper Address?
The quality of machine-generated text (MGT), from chatbots to machine translations, has improved dramatically, but it still falls short of human parity. To ensure user acceptance and further development, we need robust ways to measure this quality. While standardized methods exist for audio and visual media, the subjective quality of text, often termed "Quality of Experience" (QoE), has been largely overlooked. Existing automatic metrics for MT, like BLEU or METEOR, are fast and cheap but lack diagnostic power because they don't capture underlying quality dimensions. This paper aims to bridge this gap by identifying these crucial perceptual dimensions for MGT, specifically focusing on machine translation, as a foundational step towards building predictive models for MGT quality.
Methods
The researchers employed a two-stage approach. First, they conducted a preliminary study with 14 participants (mostly linguistic experts) to refine a set of 38 polar adjective pairs designed to capture various aspects of text quality. This involved rating the suitability of each pair and removing those with low consensus. This process reduced the adjective pairs to 20.
The core of their methodology was a large-scale crowdsourcing study involving 141 participants. They used a Semantic Differential (SD) scale, a standard psychometric method for measuring attitudes, where participants rated 45 carefully selected German MT outputs on the 20 refined adjective pairs. The participants were instructed to focus solely on the language quality, ignoring the content.
Following data collection, they performed Exploratory Factor Analysis (EFA) using SPSS. EFA is a statistical technique used to identify underlying latent variables (factors) that explain the correlations among a set of observed variables (the adjective ratings). They used Maximum Likelihood for extraction and PROMAX rotation to identify non-orthogonal dimensions. This analysis aimed to group the adjective pairs into meaningful underlying quality dimensions.
Datasets
The study used a corpus of English-to-German translations from the submissions to the News translation task of the Fourth Conference on Machine Translation (WMT19). They specifically selected translations from six systems that ranked at the top, middle, and bottom of the competition, resulting in a dataset of 11,922 sentences. A linguistic expert then curated a sub-corpus of 45 sentences from this larger set, ensuring a variety of lengths, qualities, and error types for the survey.
Key Insights and Ideas
- Beyond Fluency and Adequacy: Traditional MT quality metrics like fluency and adequacy are no longer sufficient. The study highlights that while neural MT has become more fluent, errors are subtler, necessitating a deeper understanding of perceptual quality.
- The Four Pillars of MT Quality: The EFA revealed four distinct perceptual dimensions that significantly influence the Quality of Experience (QoE) for machine-translated text:
- Precision: This dimension captures the clarity and completeness of the text.
- Complexity: This relates to the textual complexity, with longer and more convoluted sentences often scoring higher on complexity.
- Grammaticality: This dimension directly addresses grammatical correctness, including spelling and punctuation. Interestingly, the authors found that shorter sentences were more prone to grammatical errors, possibly because longer sentences can mask errors more effectively.
- Transparency: This dimension refers to the lucidity or understandability of the text, showing a notable correlation with precision.
- The "WEIRD" Bias: The authors acknowledge that their findings are based on German native speakers, potentially introducing a demographic bias (WEIRD: Western, Educated, Industrialized, Rich, and Democratic participants). This means the identified dimensions might not generalize perfectly to other languages or participant groups.
Key Results
The Exploratory Factor Analysis successfully identified four distinct quality dimensions, explaining a significant portion of the variance in the participants' ratings:
- Factor 1 (Precision): Explained 53.2% of the variance and included adjective pairs like "precise – vague" and "complete – incomplete."
- Factor 2 (Complexity): Explained 8.4% of the variance, with pairs such as "simple – complicated."
- Factor 3 (Grammaticality): Explained 10.5% of the variance, represented by "grammatical – ungrammatical."
- Factor 4 (Transparency): Explained 8.0% of the variance, with pairs like "clear – chaotic."
The study also noted that the adjective pair "fluent – non-fluent" had to be removed due to discriminant validity issues, reinforcing the idea that fluency is less of a primary concern for modern MT systems compared to other aspects.
Limitations and Future Work
The authors acknowledge several limitations:
- Data Dependency: The identified dimensions are dependent on the specific test sentences and adjective pairs used in the study.
- WEIRD Bias: The reliance on German native speakers may limit the generalizability of the findings to other linguistic and cultural contexts.
- Potential Overlap: While the four dimensions are distinct, there might be overlap between precision and transparency, as indicated by their correlation.
Future work outlined includes:
- ATS Analysis: Conducting a similar crowdsourcing study to identify quality dimensions for Automatic Text Summarization (ATS).
- Simplified Surveys: Developing simplified surveys with one representative antonym pair per dimension to verify the findings.
- Prediction Models: Correlating automatically extractable text parameters with the identified quality dimensions to build predictive models for MGT quality.
- Cross-Lingual Analysis: Expanding the analysis to other languages to counteract the WEIRD bias and explore potential cross-lingual differences in perceived quality.
- MQM Integration: Analyzing the overlap between the identified dimensions and existing quality metrics like MQM.
Summary updated