BATS: BenchmArking Text Simplicity 🦇

Christin Kreutz, Fabian Haak, Björn Engelmann, Philipp SchaerACL 2024 · DOI

TL;DR

This paper introduces BATS (BenchmArking Text Simplicity), a novel, explainable, and reference-free method for evaluating text simplicity. BATS leverages linguistic rules from literature to create "BATS vectors" that capture text characteristics, outperforming state-of-the-art methods like OpenAI embeddings in predicting human judgments of simplicity.

What Problem Does This Paper Address?

Evaluating text simplification is crucial for making information accessible, but current methods often focus on the difference between source and simplified texts, neglecting the absolute simplicity of a text. Existing datasets are often topic-specific and don't account for different target audiences (e.g., children vs. adults) or domains (e.g., medical vs. news). This leads to a lack of explainability in simplicity assessment and a gap in understanding what truly makes text simple for diverse readers. The authors aim to fill this gap by providing a method that is domain- and audience-independent and offers explainable insights into text simplicity.

Methods

The core of BATS is a data programming approach using the Snorkel framework. The authors define 37 features from existing literature that indicate text simplicity or complexity. These features are then "parametrized" into 1,249 distinct interpretations, creating a rich set of rules. Labeling functions are developed based on these parametrizations, which generate binary vectors for any given text. These "BATS vectors" represent the presence or absence of specific simplicity-indicating characteristics. A pruning step is then applied to filter out less informative dimensions, resulting in a BATS model. This model can then be used to generate BATS vectors for new texts, which are then correlated with external simplicity scores to assess their effectiveness.

Datasets

The authors utilize several datasets for their evaluation and pruning:

  • ARTS datasets (ARTS94, ARTS300, ARTS3000): These datasets provide human and GPT-4-generated simplicity scores for texts, serving as the primary benchmark for evaluating BATS.
  • 15 publicly available parallel corpora: These include datasets like ASSET, AutoMeTS, BenchLS, Britannica, EW-SEW-Turk, HutSSF, METAeval, MTurkSF, NNSeval, OneStopEnglish, QuestEval, SemEval-2007, SimPA, and SimpEval. These datasets were used for analyzing domain- and audience-specific characteristics and for pruning the BATS model.

Key Insights and Ideas

  • Explainability is Key: BATS moves beyond black-box metrics by providing interpretable "BATS vectors" that highlight why a text is considered simple or complex, based on linguistic features.
  • Literature-Driven Features Matter: Traditional linguistic features identified in the literature (like sentence length and lexical richness) remain highly indicative of text simplicity, even when compared to more complex or newer features.
  • Domain and Audience Nuances: While general simplicity features are important, the selectivity of these features varies significantly across different datasets, target audiences, and domains, underscoring the need for context-aware simplification.
  • BATS Outperforms State-of-the-Art: BATS vectors demonstrate superior performance compared to OpenAI embeddings in predicting human judgments of text simplicity, highlighting the effectiveness of their rule-based, explainable approach.

Key Results

  • BATS vectors significantly outperform OpenAI embeddings in predicting ARTS scores (simplicity scores) across various regression models (Random Forest and Gradient Boosting). For instance, in the most important setting (training on the most data, testing on the least), BATS achieved better performance than OpenAI embeddings.
  • The study identified specific linguistic features that are consistently important for simplicity and complexity across different datasets and levels of analysis (parametrizations, interpretations, features, categories).
  • Analysis of 15 datasets revealed significant differences in how simplicity characteristics manifest across domains (e.g., news vs. encyclopedia) and target audiences (e.g., children vs. language learners).

Limitations and Future Work

  • Feature Completeness: The authors acknowledge that they may not have utilized the most meaningful parameters or interpretations, and there might be other important features not captured.
  • Meaning Preservation: The current BATS parametrization does not explicitly capture meaning preservation, which is a critical aspect of text simplification.
  • Dataset Quality and Size: The evaluation was limited by the quality and size of available datasets, especially when analyzing specific target audiences and domains.
  • Future Work: The authors suggest incorporating more features into BATS, conducting in-depth evaluations of simplification approaches, and exploring domain- and audience-specific evaluation more thoroughly. They also highlight the need for datasets that contain simplifications differing by more than just one word from the source. Additionally, they propose exploring BATS for in-context learning with LLMs and for assisted text simplification.

Summary updated

BibTeX
@inproceedings{kreutz2024bats,
  author = {Christin Kreutz and Fabian Haak and Björn Engelmann and Philipp Schaer},
  title = {{BATS: BenchmArking Text Simplicity 🦇}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.712},
}
Figure