Towards Holistic Human Evaluation of Automatic Text Simplification

Luisa Carrer, Andreas Säuberli, Martin Kappus, Sarah EblingCOLING 2024

TL;DR

This paper argues that current methods for evaluating automatic text simplification (ATS) are insufficient because they often rely on subjective ratings and don't include the actual target users. The authors propose a multi-faceted evaluation approach involving end-users, post-editors, and experts, demonstrating that different user groups perceive text quality and comprehensibility differently. This holistic approach is crucial for developing ATS systems that truly benefit their intended audience.

What Problem Does This Paper Address?

Automatic Text Simplification (ATS) aims to rewrite texts into simpler language, making them more accessible to a wider audience, particularly those with reading difficulties (e.g., people with intellectual disabilities, functional illiteracy, or L2 learners). While ATS systems have advanced, evaluating their quality remains a challenge. Current evaluation practices often rely on automatic metrics (which are known to be unreliable) or human evaluations using experts or crowdworkers who rate aspects like simplicity and fluency on Likert scales.

However, these common approaches have significant shortcomings:

  1. Lack of Target User Representation: They rarely include the actual end-users of simplified texts, whose comprehension is the ultimate goal.
  2. Limited Stakeholder Inclusion: They often exclude other crucial stakeholders like post-editors who work with simplified texts professionally.
  3. Subjectivity vs. Functionality: Perceived simplicity doesn't always equate to actual comprehensibility or the text's functional success.

This paper addresses this gap by advocating for and demonstrating a more comprehensive, multi-stakeholder evaluation framework for ATS.

Methods

The authors conducted a multi-faceted evaluation study involving three distinct groups:

  1. End-User Comprehensibility Evaluation:
    • Participants: Two groups were recruited: 18 individuals with intellectual disabilities (the primary target group for Easy Language) and 18 native German speakers without intellectual disabilities (control group).
  2. Materials: 12 German texts were used, each in three versions: original source text, manually created reference simplification, and an automatically simplified version generated by a transformer-based model.
  3. Procedure: Participants read texts and answered multiple-choice comprehension questions. They also rated the perceived difficulty of each text. Behavioral data like reading speed and response time were collected.
  4. Analysis: Bayesian one-parameter logistic item response models were used to estimate text difficulty based on comprehension and perceived difficulty ratings.
  5. Post-Editing Productivity Study:
    • Participants: Four professional German-speaking translators.
  6. Materials: Eight source texts were used.
  7. Procedure: Translators performed both manual simplification (MS) and post-editing (PE) of automatically simplified texts. Effort was quantified using keystroke logging and screen recording, measuring task duration, user events (text productions, cursor navigations, mouse events), and cognitive pauses. A pre-task questionnaire gathered background data, and a post-task questionnaire assessed perceived productivity.
  8. Analysis: Statistical analysis (Student's t-tests) compared effort measures between MS and PE tasks.
  9. Expert Evaluation:
    • Participants: Four Swiss-based German-speaking experts in Easy Language translation.
  10. Materials: Texts from the post-editing study (reference, automatically simplified, manually simplified, and post-edited versions).
  11. Procedure: Experts performed a source-based direct assessment, rating simplicity, adequacy, and fluency on 5-point scales and providing comments.
  12. Analysis: Average ratings for each text category were compared.

Datasets

The study utilized a parallel corpus of original and simplified German texts provided by a commercial provider of text simplification services. This corpus was used to train the ATS model and as the basis for the evaluation materials. No new datasets were introduced, but the study used existing texts to create a benchmark for multi-stakeholder evaluation.

Key Insights and Ideas

  • User Groups Perceive Quality Differently: The most striking insight is the significant divergence in how end-users (especially those with intellectual disabilities) and control groups perceive the difficulty and quality of simplified texts. The target group found ATS output not significantly different from the source, while the control group found it slightly more difficult.
  • Subjective Ratings Aren't Enough: Perceived difficulty ratings from end-users didn't always align with actual comprehension, highlighting the need to go beyond subjective opinions.
  • Post-Editing Effort Doesn't Necessarily Decrease: Contrary to expectations, the study found no statistically significant decrease in effort for post-editing compared to manual simplification. Translators reported issues with coherence and the need for painstaking source-target comparison, impacting productivity.
  • Expert Evaluation is Necessary but Insufficient Alone: Experts identified significant issues with automatically simplified texts, deeming them "mostly not adequate" and not conforming to Easy Language guidelines. However, their evaluations alone don't capture the end-user experience.
  • Holistic Evaluation is Key: The paper strongly advocates for combining quantitative and qualitative, objective and subjective methods across multiple stakeholder groups (end-users, post-editors, experts) for a truly comprehensive ATS evaluation.

Key Results

  • End-User Comprehension: The target group (with intellectual disabilities) showed no significant difference in comprehension difficulty between ATS output and the source text. The control group found the ATS output slightly more difficult than the source.
  • Perceived Difficulty: The target group rated ATS output as simpler than the source, while the control group found it equally difficult as the source and more difficult than the reference.
  • Post-Editing Effort: No significant difference in overall effort was found between manual simplification and post-editing. However, specific user events (like cursor navigations and mouse events) were higher in post-editing, and self-assessed productivity was lower for PE tasks.
  • Expert Ratings: Manually simplified texts received the highest average ratings for simplicity, adequacy, and fluency (4.38, 4.25, 4.25 respectively). Automatically simplified texts received the lowest ratings (3.63, 3.13, 3.75 respectively). Experts frequently commented that AS texts were "mostly not adequate" and not true Easy Language.

Limitations and Future Work

  • Limitations:
    • The heterogeneity of the target group (with intellectual disabilities) led to more uncertain results and noisier data.
  • The post-editing study had a small sample size (four participants).
  • It was not always possible to collect detailed background data on the specific Easy Language guidelines experts used.
  • Future Work:
    • The authors recommend including target readers whenever possible to assess comprehensibility.
  • They suggest not relying solely on perceived quality ratings but assessing functionality directly (e.g., via comprehension or post-editing effort).
  • Collecting qualitative data (surveys, interviews) and behavioral measurements is crucial to support quantitative findings.
  • When collecting expert ratings, it's important to clearly define the Easy Language guidelines and ask evaluators to specify which rules were not observed.
  • Future research should adopt a more human-centric and holistic approach to ATS evaluation, especially as large language models become more prevalent.

Summary updated

BibTeX
@inproceedings{carrer2024towards,
  author = {Luisa Carrer and Andreas Säuberli and Martin Kappus and Sarah Ebling},
  title = {{Towards Holistic Human Evaluation of Automatic Text Simplification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure