Automatic Metrics for Genre-specific Text Quality

Annie LouisNAACL 2012

TL;DR

This paper argues that generic text quality metrics fall short and proposes novel, genre-specific metrics for academic writing, science news, and machine-generated summaries. By focusing on linguistic aspects beyond simple readability, the authors aim to capture the nuanced "well-written" characteristic that varies across different text types.

What Problem Does This Paper Address?

Current methods for automatically assessing text quality often rely on general "readability" metrics (like Flesch-Kincaid) or coherence measures that assume a single, adult-educated audience and typically use news articles as their domain. While these approaches are useful, they fail to account for the fact that what constitutes "good writing" is highly dependent on the genre and its intended audience. For instance, an academic paper prioritizes clarity and precision for experts, while a science news article aims to engage a broader audience with interesting findings. Machine-generated text, like summaries, has its own unique challenges in maintaining readability and coherence. This paper addresses this gap by proposing a more nuanced definition of text quality that is genre-aware and considers linguistic aspects beyond basic readability and coherence.

Methods

The authors propose a new definition of "text quality" as the overall well-written characteristic of an article, considering three key aspects:

  1. Fixed Audience Level: They focus on a specific, high-competency audience (e.g., educated readers for news, experts for academic papers) to isolate textual differences rather than reader comprehension.
  2. Holistic Property: Text quality is viewed as a combination of content/topic, grammaticality, discourse coherence, and writing style (e.g., use of metaphors, humor).
  3. Genre-Specific Dimensions: The importance of these factors varies by genre.

Their approach involves developing and evaluating novel metrics tailored to specific genres:

  • Academic Writing & Science Journalism: They hypothesize that these genres have distinct communicative goals and structures. They propose metrics based on:
  • Patterns in Communicative Goals: Analyzing the sequence and size of "semantic zones" (e.g., motivation, results) within texts, using syntactic patterns as a proxy for these goals.
  • General-Specific Nature of Sentences: Developing a classifier to distinguish between general and specific sentences and analyzing their balance and sequence, as summaries often need to generalize.
  • Automatic Summarization: They investigate "information cohesiveness" and "focus" as key quality aspects. They propose metrics that can predict the difficulty of source documents for summarization based on word distribution and divergence from random text.

They also consider generic features from prior work and evaluate how genre-specific metrics complement them.

Datasets

The paper mentions the following datasets and data sources:

  • Automatic Summarization: Large-scale datasets from NIST evaluation workshops, including human-rated summaries and their source documents.
  • Academic Writing: A collection of biology journal articles, using journal impact factor as a proxy for quality (though acknowledged as not a direct measure). For some experiments, they create "incoherent" versions of articles by randomly permuting sentences.
  • Science News: They plan to use Amazon Mechanical Turk for obtaining ratings and also consider proxies like lists of most emailed/viewed articles from news websites.
  • Conference Publications: ACL anthology for evaluating communicative goal metrics.
  • News Summaries: A corpus of human-written and automatically generated news summaries, along with human ratings.

Key Insights and Ideas

  • "Well-written" is Subjective and Genre-Dependent: The core idea is that a single definition of text quality is insufficient. What makes an academic paper "good" is different from what makes a science news article or a summary "good."
  • Beyond Readability: The paper moves beyond simple readability scores to capture more subtle linguistic nuances that contribute to perceived quality.
  • Communicative Goals as Structure: Identifying and analyzing the sequence of communicative goals within academic and science texts can be a strong indicator of coherence and quality.
  • Specificity Matters Differently: While general sentences are preferred for topic orientation in summaries, they need to be properly substantiated by specific details, a balance that automatic systems struggle with.
  • Source Document Difficulty: The cohesiveness and topic diversity of source documents significantly impact the quality of automatically generated summaries.

Key Results

While the paper focuses on proposing and developing metrics, it reports some preliminary results:

  • Communicative Goal Metrics: Achieved accuracies of 64-74% in distinguishing original introductions/abstracts/related work sections from randomly permuted (incoherent) versions, significantly above a 50% baseline.
  • General-Specific Sentence Classifier: Achieved an accuracy of about 79% in distinguishing general from specific sentences in news articles.
  • Specificity and Summary Quality: Human-written summaries tend to be more general than automatic ones, and human raters prefer more general content for quality in summaries, though linguistic quality scores were higher for more specific summaries.
  • Information Cohesiveness: Features like low word entropy in input documents and divergence from random text distribution were indicative of higher difficulty for summarization.

Limitations and Future Work

  • Limitations:
    • The impact factor of biology journals is an indirect measure of academic writing quality.
  • The definition of text quality is focused on discourse and style, excluding content and sentence-level issues.
  • Some proposed metrics (e.g., for science news) are still in the planning stages of evaluation.
  • Future Work:
    • Extensive evaluation of all proposed models.
  • Further investigation into style metrics for science news, including analogies, visual language, and emotional content.
  • Analyzing the relationship between cohesiveness of source texts and the "focus" aspect of automatic summaries.
  • Exploring how genre-specific metrics complement generic features across different datasets.

Summary updated

BibTeX
@inproceedings{louis2012automatic,
  author = {Annie Louis},
  title = {{Automatic Metrics for Genre-specific Text Quality}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2012},
}
Figure