Multi-Facet Counterfactual Learning for Content Quality Evaluation

Jiasheng Zheng, Hongyu Lin, Boxi Cao, Meng Liao, Yaojie Lu, Xianpei Han, Le SunarXiv 2024 · OpenReview

Abstract

Evaluating the quality of documents is essential for filtering valuable content from the current massive amount of information. Conventional approaches typically rely on a single score as a supervision signal for training content quality evaluators, which is inadequate to differentiate documents with quality variations across multiple facets. In this paper, we propose Multi-facet cOunterfactual LEarning (MOLE), a framework for efficiently constructing evaluators that perceive multiple facets of content quality evaluation. Given a specific scenario, we prompt large language models to generate counterfactual content that exhibits variations in critical quality facets compared to the original document. Furthermore, we leverage a joint training strategy based on contrastive learning and supervised learning to enable the evaluator to distinguish between different quality facets, resulting in more accurate predictions of content quality scores. Experimental results on 2 datasets across different scenarios demonstrate that our proposed MOLE framework effectively improves the correlation of document content quality evaluations with human judgments, which serve as a valuable toolkit for effective information acquisition.

Introduction

Document quality evaluation is a complex, multi-dimensional task. Traditional automated evaluators often rely on a single scalar score (e.g., 1-5) provided by human annotators. However, this approach fails to capture why a document is rated a certain way—whether due to poor coherence, lack of creativity, or low engagingness. Furthermore, quality standards are highly context-dependent; a high-quality blog post differs significantly from a high-quality academic essay. To address these challenges, we introduce the Multi-facet cOunterfactual LEarning (MOLE) framework, which teaches models to perceive fine-grained quality nuances through contrastive pairs.

Methodology: The MOLE Framework

The MOLE framework consists of two primary stages designed to move beyond simple scalar supervision:

  1. Multi-faceted Counterfactual Data Generation: We utilize Large Language Models (LLMs), such as GPT-3.5-Turbo, to generate "what-if" scenarios. Starting with an original document and its human-assigned score, the LLM is prompted to identify specific quality facets (e.g., coherence, engagingness, creativity) and rewrite the text to either improve or degrade those specific dimensions. This process creates contrastive pairs where the primary difference is a single quality facet.
  2. Joint Learning Strategy: A base model (e.g., LLaMA2-7b-chat) is fine-tuned using a dual-objective approach. It performs standard supervised learning to predict quality grades and contrastive learning to distinguish between original documents and their counterfactual counterparts. This allows the model to learn specific linguistic markers associated with different quality levels.
Overview of the MOLE framework, illustrating counterfactual data construction and the joint learning process.

Experimental Results

We evaluated MOLE on two distinct datasets: a Web-Article quality dataset (Chinese) and the ASAP essay-scoring dataset (English).

Performance and Correlation

MOLE demonstrated superior performance compared to strong baselines like Prometheus and Auto-J. On the Web-Article dataset, MOLE achieved a Spearman correlation ($\rho$) of 0.535, a significant improvement over the base LLaMA2-Origin model (0.508). In the ASAP dataset, MOLE showed marked improvements in perceiving sub-dimensions such as "Writing Applications" and "Language Conventions."

Handling Extreme Cases

One of the most significant advantages of MOLE is its ability to identify very low-quality content (Grade 0 or 1), which is often underrepresented in training data. By explicitly generating counterfactual examples for these grades, MOLE significantly boosted the F1 score for Grade 0 articles from 0.024 to 0.106.

F1-Score comparison across different quality grades, showing MOLE's significant improvement in identifying low-quality content.

Conclusion

The MOLE framework effectively enhances content quality evaluation by incorporating multi-faceted counterfactual learning. While the approach depends on the quality of the LLM used for data generation and involves non-trivial computational costs, it offers a scalable and adaptable solution for diverse evaluation scenarios. Future work will focus on improving the robustness of the counterfactual generation process and exploring its application in more specialized domains.

Summary updated

Abstract

Evaluating the quality of documents is essential for filtering valuable content from the current massive amount of information. Conventional approaches typically rely on a single score as a supervision signal for training content quality evaluators, which is inadequate to differentiate documents with quality variations across multiple facets. In this paper, we propose Multi-facet cOunterfactual LEarning (MOLE), a framework for efficiently constructing evaluators that perceive multiple facets of content quality evaluation. Given a specific scenario, we prompt large language models to generate counterfactual content that exhibits variations in critical quality facets compared to the original document. Furthermore, we leverage a joint training strategy based on contrastive learning and supervised learning to enable the evaluator to distinguish between different quality facets, resulting in more accurate predictions of content quality scores. Experimental results on 2 datasets across different scenarios demonstrate that our proposed MOLE framework effectively improves the correlation of document content quality evaluations with human judgments, which serve as a valuable toolkit for effective information acquisition.
BibTeX
@misc{zheng2024multi,
  author = {Jiasheng Zheng and Hongyu Lin and Boxi Cao and Meng Liao and Yaojie Lu and Xianpei Han and Le Sun},
  title = {{Multi-Facet Counterfactual Learning for Content Quality Evaluation}},
  year = {2024},
  eprint = {2410.07693},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.07693},
}
Figure