TL;DR
Evaluating automatic sentence simplification (SS) systems is a complex task with fragmented tools and datasets. This paper introduces EASSE, a Python package that unifies popular SS evaluation metrics, provides access to standard datasets, and generates comprehensive HTML reports, making the evaluation process more accessible and reproducible.
What Problem Does This Paper Address?
Sentence Simplification (SS) aims to make text easier to understand while preserving its original meaning. Evaluating how well SS systems achieve this is crucial but challenging. Researchers often face a fragmented landscape of evaluation tools, with metrics implemented in different programming languages, requiring complex setups, or lacking clear documentation. This makes it difficult to compare systems fairly and reproduce results. Existing metrics also have limitations: some rely heavily on reference simplifications, while others, like readability scores, can be misleading. The authors identify a need for a standardized, user-friendly platform that consolidates various evaluation approaches, from traditional metrics to more nuanced analyses.
Methods
EASSE tackles this problem by acting as a comprehensive Python package with a command-line interface. Its core contributions include:
- Unified Metrics: It wraps popular automatic metrics like BLEU (for grammaticality and meaning preservation), SARI (for measuring changes in simplicity), and FKGL (a readability score). EASSE uses established libraries like SACREBLEU for BLEU and re-implements SARI and FKGL for consistency and ease of use.
- Word-Level Transformation Analysis: Beyond aggregate scores, EASSE analyzes specific text transformations (additions, deletions, replacements, copies) performed by SS systems. This is achieved by adapting algorithms from MASSAlign, comparing system outputs to reference simplifications to estimate transformation correctness.
- Quality Estimation (QE) Features: To address the limitations of reference-dependent metrics, EASSE incorporates reference-less QE features. These include compression ratio, Levenshtein similarity, sentence splitting proportions, and lexical complexity scores, offering insights into system behavior without needing human-written references.
- Dataset Access: EASSE provides direct access to commonly used SS evaluation datasets: PWKP, TurkCorpus, and HSplit. It also allows users to evaluate systems on custom datasets.
- HTML Report Generation: A key feature is the generation of a comprehensive, easy-to-visualize HTML report. This report consolidates all metric scores, transformation analyses, and QE features, often presented with visualizations and comparisons against references, facilitating a deeper understanding of system performance.
Datasets
EASSE provides access to the following publicly available datasets for SS evaluation:
- PWKP: Contains 65,133 English Wikipedia (EW) articles aligned to their Simple EW (SEW) counterparts. It primarily uses 1-to-1 alignments, with a small portion of 1-to-N alignments for sentence splitting.
- TurkCorpus: Built upon PWKP, this dataset features 2,359 sentences simplified by crowdworkers, with each original sentence having 8 simplified references. It's well-suited for metrics like SARI that benefit from multiple references.
- HSplit: Addresses the lack of sentence splitting instances in existing datasets by collecting four reference simplifications for sentence splitting for the 359 sentences in the TurkCorpus test set.
Key Insights and Ideas
- Beyond Aggregate Scores: The paper strongly advocates for moving beyond single-score evaluations (like BLEU or SARI alone). By combining these with word-level transformation analysis and QE features, a much richer picture of a system's strengths and weaknesses emerges.
- Complementary Evaluation: QE features are shown to be crucial for understanding system behavior that traditional metrics miss. For example, a system might have a good SARI score but a very low compression ratio, indicating it's not simplifying effectively.
- Dataset Nuances Matter: The choice of dataset significantly impacts evaluation results. TurkCorpus, with its focus on paraphrasing, favors systems good at replacements and copying, while HSplit is designed to test sentence splitting capabilities.
- Reproducibility is Key: By standardizing metrics and providing access to datasets, EASSE directly addresses the reproducibility crisis in NLP research.
Key Results
The paper presents experimental results comparing several SS systems (PBSMT-R, Hybrid, SBSMT-SARI, DRESS-LS, DMASS-DCSS) on TurkCorpus and HSplit.
- DMASS-DCSS (a Transformer-based system) achieved state-of-the-art SARI scores on TurkCorpus, indicating strong performance in simplification.
- PBSMT-R performed best on BLEU and SAMSA metrics, suggesting good grammaticality and meaning preservation, and effective structural simplification.
- Word-level analysis revealed that SBSMT-SARI excels at replacements, contributing to its high SARI score, while PBSMT-R is best at copying. Hybrid showed a high proportion of deletions but poor replacements.
- QE features highlighted that Hybrid significantly compresses text (ratio of 0.57) but rarely adds new words, explaining its high deletion rate. DRESS-LS kept source sentences unmodified 26% of the time, a detail not captured by word-level analysis alone.
Limitations and Future Work
- Limitations:
- The paper notes that at the time of submission, only a subset of HSplit sentences were available.
- Automatic word-level transformation annotation algorithms have an inherent error rate (micro-averaged F1 of 0.61 compared to manual labels).
- FKGL readability scores should be interpreted with caution as they can be misleading.
- Future Work:
- Developing more sophisticated transformation-based analysis algorithms to identify complex transformations like subject-verb-object reordering.
- Integrating additional QE features to capture other aspects of simplification, such as syntactic complexity (e.g., dependency parse tree depth).
Summary updated