TL;DR
This paper tackles the challenge of producing high-quality text simplifications, which are currently too expensive to generate with human experts. They introduce a novel sentence fusion technique to create more simplification candidates from crowdsourced efforts and a reranking approach to select the best ones, achieving better simplicity and adequacy than existing human baselines.
What Problem Does This Paper Address?
Automated text simplification systems, while improving, still struggle to meet the quality standards required for real-world applications, especially in sensitive domains like health and medicine. Consequently, human experts are still the primary source of simplified text. However, employing human experts is costly and time-consuming, and the pool of skilled individuals is limited. This paper explores how to leverage crowdsourcing, using non-expert annotators, to produce simplifications more efficiently and with higher quality. The core challenge is to control the quality of these crowdsourced simplifications and to effectively target different levels of simplicity.
Methods
The authors propose a two-pronged approach:
- Sentence Fusion: They introduce a graph-based sentence fusion technique. This method constructs a graph where nodes represent words and edges represent possible transitions between them. Sentences (both original and crowdsourced simplifications) are added to this graph, creating new paths that represent novel, fused simplifications. To improve the graph's quality and reduce spurious connections, they employ word alignment between crowdsourced simplifications and the original sentence, considering words equivalent only if they are lexically identical and aligned.
- Reranking: A supervised, feature-based pairwise ranking approach is used to select the best simplification from a pool of candidates. This pool includes the original sentence, the crowdsourced simplifications, and the newly generated sentence fusion candidates. The reranker is trained to optimize a composite score that balances simplicity and adequacy, using a weighted geometric mean. They also introduce a parameter, $\alpha$, to bias the ranking towards either simplicity or adequacy.
Datasets
- Newsela Corpus: This is the primary dataset used for both training and evaluation. It's a sentence-aligned corpus derived from articles simplified by experts at four distinct simplicity levels (V1-V4). The authors note that this dataset provides a strong baseline with expert simplifications and is suitable for testing their target-simplicity-specified rerankers.
- Crowdsourced Data: The authors collected their own crowdsourced simplifications and human evaluations (simplicity, adequacy, fluency) using Amazon Mechanical Turk. This data was used to train their reranking model.
Key Insights and Ideas
- Crowdsourcing for Quality: The paper demonstrates that by intelligently combining and reranking outputs from non-expert crowdsourced annotators, it's possible to achieve simplification quality comparable to or better than expert-generated simplifications.
- Sentence Fusion as a Candidate Generator: Sentence fusion is a powerful way to generate a much larger and more diverse set of simplification candidates than relying solely on individual crowdsourced outputs. This broadens the search space for high-quality simplifications.
- Controllable Simplicity: The introduction of the $\alpha$ parameter in the reranking objective allows for fine-grained control over the desired level of simplicity in the output, enabling targeted simplification.
- Simplicity-Adequacy Tradeoff: The work explicitly addresses the inherent tradeoff between making text simpler and retaining its original meaning (adequacy), proposing a method to balance these competing goals.
Key Results
- The proposed system consistently outperforms the human expert baseline (Newsela) at various simplicity levels in terms of both simplicity and adequacy.
- For a specific setting of $\alpha$ (balancing simplicity and adequacy), the system produces simplifications that are significantly simpler and more adequate than comparable Newsela versions.
- The sentence fusion approach contributes significantly, especially when simplicity is prioritized, allowing for simpler outputs than human-only simplifications.
- The system generates fusion-generated sentences for 30-40% of the simplifications when balancing adequacy and simplicity.
Limitations and Future Work
- Reranker Improvement: The authors acknowledge that the reranker can still be improved, as suggested by the performance gap between their reranked results and an "oracle" study (where the best candidate is known).
- Domain Adaptation: The paper raises the question of how well the trained reranker generalizes to different domains, suggesting this as a future research direction.
- Parameter Tuning: Further research is needed to establish a standard for choosing appropriate values of $\alpha$ based on the relationship between $\alpha$ and simplicity.
- Alternative Fusion Techniques: While they used a graph-based approach, the authors suggest that recent neural approaches from abstractive summarization could be adapted for sentence fusion.
- Feature Exploration: Additional features and alternative reranking algorithms could further enhance performance.
Summary updated