Adapting Sentence-level Automatic Metrics for Document-level Simplification Evaluation

Mounica Maddela, Fernando Alva-ManchegoNAACL 2025 · DOI

Introduction

Evaluating document-level text simplification (DTS) is a significant challenge in natural language processing. Unlike sentence-level simplification, which often focuses on lexical substitution, DTS involves complex structural changes such as sentence reordering, splitting, fusion, and selective deletion. Current automatic metrics like SARI or LENS are primarily optimized for single-sentence pairs and struggle to capture these "many-to-many" relationships. Furthermore, many modern transformer-based metrics are limited by input length constraints, making them unsuitable for long documents.

The Challenge of Document-Level Evaluation

Standard metrics face two primary hurdles when applied to documents:

  1. Granularity and Alignment: Metrics often assume a linear 1:1 mapping between source and simplified sentences. In reality, a single complex sentence might be split into several simpler ones, or multiple sentences might be fused to improve flow.
  2. Technical Constraints: Models like RoBERTa, which underpin metrics like LENS, typically have a 512-token limit, preventing them from processing entire documents as a single input.

Methodology: A Divide-and-Conquer Framework

To address these issues, we propose a metric-agnostic aggregation framework that breaks long documents into manageable, semantically related "mini-units."

  1. Similarity Matrix Construction: We utilize a sentence-pair similarity model to compare every sentence in the source text against the simplified and reference versions.
  2. Graph-Based Alignment: We construct an undirected graph where sentences are nodes. Edges are created between sentences with a similarity score above 0.5. By identifying the "connected components" of this graph, we group related sentences across all versions of the text, effectively capturing many-to-many transformations.
  3. Metric Aggregation: Existing metrics (e.g., LENS, SARI, BERTScore) are applied to these aligned units. The resulting scores are averaged to produce a final document-level evaluation.

Experimental Results

We benchmarked our "Agg-metrics" against traditional sentence-level and document-level baselines across three datasets: COCHRANE (medical abstracts), D-WIKIPEDIA (encyclopedic entries), and ONESTOPQA (news articles).

Correlation of automatic metrics with human judgments across three datasets.

As shown in the results table, Agg-LENS consistently outperforms other metrics. On the Cochrane dataset, it achieved a Kendall Tau correlation of 0.433, a dramatic improvement over the document-level baseline DSARI (-0.017).

Scaling with Document Length

A key advantage of the aggregation approach is its robustness as document length increases. While standard sentence-level metrics often see a decline in performance on longer texts, Agg-LENS maintains or even improves its correlation with human judgment as the number of tokens grows.

Kendall Tau-like correlation of Agg-LENS, LENS, and DSARI as a function of source document length.

Discussion and Limitations

While aggregation improves sensitivity to structural changes and readability, it introduces a trade-off regarding "hallucinations"—cases where the model adds information not present in the source. Because scores are averaged across multiple units, a single unit containing a hallucination may be "diluted" by other correctly simplified units, making the metric less likely to penalize the error heavily.

Furthermore, our current framework is optimized for English and focuses on transformations like splitting and fusing. Future work will explore "elaborative simplification," where explanatory content is added, and aim to increase metric sensitivity to factual inconsistencies.

Conclusion

By adapting high-performing sentence-level metrics through a graph-based alignment strategy, we provide a more reliable way to evaluate document-level simplification. Agg-LENS, in particular, emerges as a robust tool for assessing readability and structural transformation in complex, long-form text.

Summary updated

BibTeX
@inproceedings{maddela2025adapting,
  author = {Mounica Maddela and Fernando Alva-Manchego},
  title = {{Adapting Sentence-level Automatic Metrics for Document-level Simplification Evaluation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.327},
}
Figure