Joint Sentence and Word Level Quality Estimation

102 concepts71 papers1 proposal

When evaluating the output of generative models—whether a machine translation, an automated essay, or a speech transcript—we often ask the model to produce a single, holistic quality score. However, mapping a complex sequence of text to a single scalar value is a difficult optimization problem. A model trained solely on global scores lacks explicit guidance on why a sequence is poor. It might penalize a translation heavily for a single critical error (like a flipped negation) or fail to notice a "word salad" that contains sophisticated vocabulary but lacks structural coherence.

The core insight of multi-task quality estimation is that global quality is an aggregate of local correctness. Instead of treating quality estimation as a monolithic regression problem, modern architectures jointly optimize for multiple granularities simultaneously: sentence-level scores, word-level error tags, and trait-specific attributes. This multi-task approach forces the model's internal representations to ground abstract quality metrics in concrete, localized evidence.

Anchoring Global Scores in Local Evidence

Why does training a model to predict word-level error tags improve its ability to predict a sentence-level score? In a standard regression setup, a model can easily overfit to surface-level patterns or become uncalibrated across different domains. By adding an auxiliary sequence-labeling task, the model must explicitly identify which specific tokens are incorrect, redundant, or missing.

As shown in the Findings of the WMT 2020 Shared Task on Quality Estimation by Marina Fomicheva et al. and the Findings of the WMT 2023 Shared Task on Quality Estimation by Chrysoula Zerva et al., this joint training acts as a powerful regularizer. The fine-grained word-level signals provide an anchoring mechanism that ensures sentence-level scores remain consistent with detected token-level defects.

Architectures like QE BERT: Bilingual BERT Using Multi-task Learning for Neural Quality Estimation by Hyun Kim et al. and CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task by Marcos Vinicius Treviso et al. implement this by sharing a multilingual encoder across both tasks. The token representations are fed into a sequence-tagging head to classify individual word errors, while an aggregated sequence representation is used for global score regression. This design ensures that the model cannot easily assign a high quality score to a sentence containing critical token-level failures.

Furthermore, as demonstrated by Ricardo Rei et al. in COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task and Ana C. Farinha et al. in Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task, this dual-task approach provides built-in explainability, allowing users to see exactly which words triggered a low score. Xiang Geng et al. further unified this in Unify Word-level and Span-level Tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task by showing that shared representations allow models to seamlessly map token-level probabilities to fine-grained error spans. Even when distilling massive models into smaller, efficient variants, Daniil Larionov et al. demonstrated in xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics that retaining the word-level error span prediction task is vital for preserving the metric's interpretability and accuracy.

Cross-Granularity Generalization Beyond Translation

The principle of multi-granularity joint training extends far beyond machine translation. In Automated Essay Scoring (AES), a single holistic grade often obscures specific writing flaws. To combat this, Zixuan Ke et al. developed an architecture in Learning to Give Feedback: Modeling Attributes Affecting Argument Persuasiveness in Student Essays that simultaneously predicts a holistic quality score and multiple fine-grained descriptive attributes (such as eloquence and evidence quality). This multi-task framework forces the model to correlate its final output with explicit qualitative criteria.

Similarly, Shengjie Li 0002 et al. noted in Automated Essay Scoring: Recent Successes and Future Directions that predicting trait-specific scores alongside a total score drastically improves feedback interpretability. To prevent models from being fooled by adversarial inputs, Youmna Farag et al. proposed in Neural Automated Essay Scoring and Coherence Modeling for Adversarially Crafted Input a multi-task architecture that couples global quality assessment with a sliding-window branch for local sentence coherence, ensuring that logical transitions directly inform the final score.

In speech processing, a similar hierarchy applies. David Qiu et al. introduced a framework in Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction that jointly optimizes word-level correctness, utterance-level accuracy, and explicit deletion counts. By sharing signals across these granularities, the model produces highly calibrated confidence scores. For open-response pronunciation assessment, Julia Hirschberg et al. developed MultiPA: A Multi-task Speech Pronunciation Assessment Model for Open Response Scenarios, which simultaneously predicts sentence-level fluency and word-level phonetic accuracy, leveraging shared representations to ensure local acoustic details inform global prosody evaluations.

Even in structured semantic domains or video analysis, the pattern holds:

Mitigating Over-Correction in Generation and Editing

Multi-task quality estimation does not just evaluate text; it also acts as a critical constraint during text generation and error correction. A common failure mode of sequence-to-sequence correction models (like Grammatical Error Correction or Automatic Post-Editing) is over-correction—the tendency to rewrite perfectly valid text because the model is biased toward modification.

By integrating error detection directly as an auxiliary task within the generation model, the system learns exactly where to edit and, crucially, where to leave the text untouched. Wei Zhao et al. demonstrated this in Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data, where token-level error detection tasks forced a copy-augmented Transformer to remain conservative, preventing unnecessary changes to correct text. Similarly, Zheyu He et al. developed UMRSpell: Unifying the Detection and Correction Parts of Pre-trained Models towards Chinese Missing, Redundant, and Spelling Correction to unify error detection and correction into a single multi-task model, allowing the network to learn the explicit relationship between identifying an error and fixing it simultaneously.

In Automatic Post-Editing (APE), Diptesh Kanojia et al. showed in Quality Estimation-Assisted Automatic Post-Editing that tightly coupling sequence-to-sequence correction with sentence- and word-level quality estimation prevents over-correction. The quality estimation task acts as a regularizing constraint that tells the correction mechanism exactly which segments require modification. This was extended to low-resource multilingual settings by Diptesh Kanojia et al. in Together We Can: Multilingual Automatic Post-Editing for Low-Resource Languages, proving that predicting word-level quality scores alongside text generation prevents the over-correction of accurate segments across related languages.

Unifying Evaluation Paradigms

Finally, multi-task learning allows models to bridge disparate evaluation methodologies and objectives. Traditionally, quality estimation (reference-free) and metric evaluation (reference-based) were treated as separate problems requiring separate models.

In UniTE: Unified Translation Evaluation, Yu Wan demonstrated that training a single model on multiple evaluation paradigms—source-only, reference-only, and combined source-reference inputs—enables the model to learn shared underlying features of translation quality, facilitating cross-task knowledge transfer. David Ifeoluwa Adelani et al. leveraged this in AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages, showing that combining reference-based scoring and reference-free quality estimation within a single multi-task model yields superior performance for under-resourced languages by better capturing semantic adequacy when references are missing.

Furthermore, models can simultaneously optimize for different mathematical formulations of quality. Jingyi Zhang et al. showed in Translation Quality Estimation by Jointly Learning to Score and Rank that a multi-task framework optimizing for both absolute quality scores (regression) and relative preference rankings (pairwise classification) improves score calibration. By learning from both direct assessments and pairwise comparisons, the model maintains a clear distinction between translations of varying quality while outputting stable absolute scores.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_task_quality_estimation_a8b7a1ec,
  title = {multi-task quality estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-task-quality-estimation/rdr_a8b7a1ec80234322b401a7b6b3834295},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_task_quality_estimation_a8b7a1ec_v1,
  title = {multi-task quality estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-task-quality-estimation/rdr_a8b7a1ec80234322b401a7b6b3834295/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake