Abstract
Semantic Role Labeling (SRL) is a critical task in natural language processing that identifies the underlying predicate-argument structure of sentences. However, SRL models often struggle with data sparsity due to the vast syntactic variety in natural language. This paper presents a data augmentation approach that utilizes automatic text rewriting to generate syntactic variants of existing training data. By extracting rewrite rules from Wikipedia revision histories and paraphrase databases, and applying a quality-control refinement filter, the researchers significantly improved performance on the CoNLL-2009 benchmark. The results demonstrate that exposing models to diverse syntactic expressions of the same semantic content enhances both in-domain accuracy and out-of-domain robustness.
Introduction
Semantic Role Labeling (SRL) aims to answer the fundamental question of "who did what to whom, when, and where" within a sentence. For instance, in the sentence "The chef cooked the meal," the chef serves as the Agent and the meal as the Theme. Despite its importance, SRL faces a significant hurdle: data sparsity. Models rely heavily on the syntactic path between a verb and its arguments, but English offers numerous ways to express the same meaning—through passive voice, appositives, or complex nested clauses. Because human-annotated data is expensive, models frequently encounter unfamiliar syntactic structures in real-world applications.
Methodology: Data Augmentation via Text Rewriting
To address this, researchers at the University of Edinburgh proposed a pipeline that expands small "gold standard" datasets using automatic text rewriting.
Rule Extraction
Rewrite rules were automatically extracted from three primary sources:
- Wikipedia Revision Histories: Capturing how editors simplify or reorder sentences.
- Simple vs. Standard Wikipedia: Identifying simplification patterns across parallel articles.
- ParaPhrase DataBase (PPDB): Utilizing bilingual pivoting (e.g., English to German and back) to discover lexical and phrasal paraphrases.
The Refinement Filter and Label Projection
Not all rewrites are beneficial; some may compromise grammatical integrity or semantic meaning. The authors implemented a "quality control" step using a binary Support Vector Machine (SVM) classifier to predict which rules were "label-preserving." A rule was deemed trustworthy if an initial SRL model could still correctly identify roles in the rewritten output. Once validated, the original "gold" labels were projected onto the new sentence structures, and the model was retrained on the combined dataset.
Experimental Results
The study utilized the CoNLL-2009 Shared Task benchmark for evaluation. Key findings include:
- State-of-the-Art Performance: Models trained on rewritten data significantly outperformed previous systems. On the in-domain test set, the best combined model achieved an F1 score of 85.71, a statistically significant improvement over the 85.15 baseline.
- Out-of-Domain Robustness: Performance gains were even more pronounced on out-of-domain data (increasing from 73.82 to 74.83 F1), indicating that syntactic variety aids generalization across different writing styles.
- Recall Improvements: The most substantial gains were seen in recall, suggesting the model became more adept at identifying arguments in complex or unusual configurations.
Discussion: The Power of Syntactic Gymnastics
The core insight of this work is that models benefit less from simply more data and more from seeing the same semantic relationships expressed through different "syntactic gymnastics." By providing variants, the feature space for the classifier is effectively smoothed. Interestingly, automatically learned rules from Wikipedia outperformed sophisticated hand-crafted rules, as they captured a broader spectrum of natural linguistic shifts.
Conclusion
This research demonstrates that automatic text rewriting is a powerful, cost-effective method for boosting SRL performance. By teaching models to recognize semantic roles across diverse syntactic structures, the approach mitigates the bottleneck of human annotation. Future work may explore more sophisticated filtering techniques, such as n-gram language models for fluency, and the application of these methods to unsupervised semantic role induction or machine translation.
Summary updated