Automating Easy Read Text Segmentation

Jesus Calleja, Thierry Etchegoyhen, Antonio David Ponce MartínezEMNLP 2024 · DOI

Introduction

Easy Read (ER) is a specialized adaptation style designed to make information accessible to people with cognitive disabilities. Beyond simplifying vocabulary and syntax, ER relies heavily on visual layout: sentences are broken into separate lines at natural linguistic boundaries where a reader would typically pause. While automated text simplification has seen significant progress, the task of automated segmentation—determining where these line breaks should occur—has remained largely unexplored. This research presents the first comprehensive study on automating ER text segmentation across three languages: English (EN), Spanish (ES), and Basque (EU).

Methodology

The researchers evaluated two primary technical approaches to segmentation:

1. Unsupervised Scoring-based Segmentation

This approach uses existing linguistic tools to score potential break points without requiring explicit training on ER data:

  • Constituency Parsing: Using tools like Stanza to measure the "distance" between words in a syntax tree. A larger distance typically indicates a more natural break point.
  • Masked Language Models (MLM): Using BERT-style models to predict the likelihood of a punctuation mark appearing after a word. This serves as a proxy for identifying natural pauses.
  • Beam Search: To ensure segments are of a readable length (ideally 5–15 words), a beam search algorithm finds the optimal path of segments based on the model's scores.

2. Generative LLM Segmentation

The study also tested the capabilities of Large Language Models (LLMs) like Llama2-7B and GPT-4. These were evaluated in zero-shot and few-shot configurations, as well as through fine-tuning using Low-Rank Adaptation (LoRA) on specialized ER datasets.

Datasets

To address the lack of benchmarks, the authors curated new ER segmentation datasets for English, Spanish, and Basque. These datasets pair original sentences with expert-crafted, line-broken versions. The inclusion of Basque is particularly noteworthy as it is an agglutinative, head-final language, providing a rigorous test for the cross-linguistic generalizability of these models.

Results and Performance Analysis

The experimental results highlighted a significant trade-off between naturalness and text preservation.

Table 3: Performance metrics across models and languages. MLM scoring consistently provides the best balance of text preservation (BLEU) and segmentation accuracy (F1).

As shown in the results table, MLM-based scoring emerged as the most reliable performer. It achieved the highest F1 scores (ranging from 40.80 to 42.16) while maintaining a perfect BLEU score of 100.00, meaning it never altered the original text. In contrast, generative LLMs like GPT-4 often "hallucinated" changes to the text even when instructed only to segment it, leading to lower F1 scores despite high "Sigma" scores (a metric for naturalness). Fine-tuning Llama2 (Latxa-FT) significantly improved its performance over zero-shot versions, bringing its naturalness scores close to GPT-4 while better preserving the input text.

Human Evaluation

A qualitative human evaluation was conducted to compare machine-generated segments against expert human references.

Table 6: Human evaluation results comparing MLM-based segmentation with expert human segmentation.

The evaluation revealed that while the MLM system is highly effective, a gap remains. Human experts were rated higher for "respecting reading pauses" (~70% vs ~60%) and "grammatical boundaries." Interestingly, the evaluation also highlighted the subjectivity of the task: evaluators only agreed with the "gold standard" human references 50-80% of the time, suggesting that there is often more than one "correct" way to segment a sentence for Easy Read.

Conclusion and Future Directions

This study demonstrates that automated ER segmentation is feasible, with unsupervised MLM-based scoring providing a stable and effective baseline. While generative LLMs show promise in capturing natural pauses, their tendency to rephrase text remains a hurdle for pure segmentation tasks.

Future work should address the "under-splitting" problem, where models leave segments too long, and improve the handling of complex named entities. Most importantly, future evaluations should involve the actual target audience—individuals with cognitive disabilities—to directly measure the impact of automated segmentation on reading comprehension and ease.

Summary updated

BibTeX
@inproceedings{calleja2024automating,
  author = {Jesus Calleja and Thierry Etchegoyhen and Antonio David Ponce Martínez},
  title = {{Automating Easy Read Text Segmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.694},
}
Figure