TL;DR
This paper introduces a novel non-autoregressive model for controllable text simplification, allowing users to specify a target reading grade level. By integrating lexical complexity signals into an edit-based approach, their model generates simplifications that better match the desired complexity than traditional autoregressive methods, while also enabling more nuanced control over the simplification process.
What Problem Does This Paper Address?
Text simplification (TS) aims to make complex text easier to understand. However, "simplicity" is subjective and depends on the target audience. Existing TS models often generate a single "generic" simplified output, failing to cater to specific needs, such as different reading grade levels. While some work has attempted to control simplification for a target grade level, this control is often coarse. This paper addresses the need for Controllable Text Simplification (CTS), where the system can adapt text to a specific reading grade level (e.g., for K-12 students) with finer-grained control over the simplification process. The challenge lies in generating simplifications that are not only grammatically correct and preserve meaning but also precisely match the target complexity without over- or under-simplifying.
Methods
The authors propose a non-autoregressive (NAR) edit-based approach, repurposing a model called EDITOR (originally for machine translation) for CTS. EDITOR works by iteratively refining an input sequence through a series of edit operations: repositioning (changing token order, including deletions) and insertion (adding new tokens).
Here's what's novel:
- Controllable Input: The target grade level is encoded as a special control token, guiding the model's generation.
- Lexical Complexity Integration: Crucially, they identify words in the source text that are too complex for the target grade level using Pointwise Mutual Information (PMI) and Automatic Readability Index (ARI). These complex words are used to create an initial sequence for EDITOR, effectively pre-simplifying the input. This provides a more direct signal than just a grade-level token.
- End-to-End Training: Unlike some prior edit-based models that use pipelines, their approach is trained end-to-end, allowing it to learn sequences of edits for complex operations like sentence splitting and paraphrasing.
Datasets
The primary dataset used is Newsela-Grade, a corpus of news articles rewritten by professional editors for different grade levels. This dataset is well-suited for CTS as it provides parallel texts at various complexity levels. The authors use a subset of this data, pre-processed with Moses tools and subword tokenization.
Key Insights and Ideas
- Lexical Complexity as an Initializer: Using lexical complexity signals (PMI, ARI) to create an initial, partially simplified sequence for an edit-based model is more effective than relying solely on a target grade-level token. It provides a stronger, more granular starting point for refinement.
- Non-Autoregressive Edit-Based Models Excel at CTS: The iterative, edit-based nature of the NAR model allows it to perform complex operations like sentence splitting and paraphrasing more naturally than autoregressive models, which might struggle to "copy" these operations implicitly.
- Fine-tuning is Key: The paper highlights that fine-tuning the EDITOR model for refinement, informed by the initial lexical complexity, is crucial for achieving good performance, rather than just using it for generation from scratch.
- Trade-offs in Simplification: While their model achieves better grade-level matching, human evaluation suggests a slight trade-off in meaning preservation compared to autoregressive baselines, indicating an area for future improvement.
Key Results
- Automatic Metrics: The proposed NAR approach significantly outperforms autoregressive (AR) baselines on key metrics like SARI (lexical simplification), PCC (correlation with reference complexity), and ARI Accuracy (matching target grade level). For example, their approach achieved a SARI of 39.1 ±0.1 compared to the best AR baseline's 38.7 ±0.5.
- Human Evaluation: Human evaluators rated the system's outputs as closer to the reference in terms of simplicity compared to the AR baseline.
- Edit Operations: The model effectively learns to perform complex edits like sentence splitting and lexical substitution sequentially, which are often challenging for other methods.
- Performance across Grades: The model performs comparably or better than AR baselines across most grade levels, with slight dips at very low grades (2-3) where aggressive deletion might occur.
Limitations and Future Work
- Grammaticality: The authors acknowledge that their model's outputs are sometimes less grammatical than AR models, potentially due to the independence assumptions in NAR models.
- Meaning Preservation: Human evaluation indicated a slight decrease in meaning preservation compared to AR baselines, suggesting a trade-off that needs further balancing.
- Content Elaboration: The model struggles with generating elaborations and explanations, which would require knowledge-based reasoning beyond its current capabilities.
- Future Directions: The authors suggest improving grammaticality while maintaining tight control over complexity by better aligning atomic edit operations with more complex simplification strategies. They also note that the iterative process is more beneficial when the gap between source and target grades is wider.
Summary updated