Text Rewriting Improves Semantic Role Labeling

Kristian Woodsend, Mirella Lapata

Abstract

Semantic Role Labeling (SRL) is a critical task in natural language processing that identifies the underlying predicate-argument structure of sentences. However, SRL models often struggle with data sparsity due to the vast syntactic variety in natural language. This paper presents a data augmentation approach that utilizes automatic text rewriting to generate syntactic variants of existing training data. By extracting rewrite rules from Wikipedia revision histories and paraphrase databases, and applying a quality-control refinement filter, the researchers significantly improved performance on the CoNLL-2009 benchmark. The results demonstrate that exposing models to diverse syntactic expressions of the same semantic content enhances both in-domain accuracy and out-of-domain robustness.

Introduction

Semantic Role Labeling (SRL) aims to answer the fundamental question of "who did what to whom, when, and where" within a sentence. For instance, in the sentence "The chef cooked the meal," the chef serves as the Agent and the meal as the Theme. Despite its importance, SRL faces a significant hurdle: data sparsity. Models rely heavily on the syntactic path between a verb and its arguments, but English offers numerous ways to express the same meaning—through passive voice, appositives, or complex nested clauses. Because human-annotated data is expensive, models frequently encounter unfamiliar syntactic structures in real-world applications.

Methodology: Data Augmentation via Text Rewriting

To address this, researchers at the University of Edinburgh proposed a pipeline that expands small "gold standard" datasets using automatic text rewriting.

Rule Extraction

Rewrite rules were automatically extracted from three primary sources:

  1. Wikipedia Revision Histories: Capturing how editors simplify or reorder sentences.
  2. Simple vs. Standard Wikipedia: Identifying simplification patterns across parallel articles.
  3. ParaPhrase DataBase (PPDB): Utilizing bilingual pivoting (e.g., English to German and back) to discover lexical and phrasal paraphrases.
Examples of text rewriting transformations used for data augmentation.

The Refinement Filter and Label Projection

Not all rewrites are beneficial; some may compromise grammatical integrity or semantic meaning. The authors implemented a "quality control" step using a binary Support Vector Machine (SVM) classifier to predict which rules were "label-preserving." A rule was deemed trustworthy if an initial SRL model could still correctly identify roles in the rewritten output. Once validated, the original "gold" labels were projected onto the new sentence structures, and the model was retrained on the combined dataset.

Experimental Results

The study utilized the CoNLL-2009 Shared Task benchmark for evaluation. Key findings include:

  • State-of-the-Art Performance: Models trained on rewritten data significantly outperformed previous systems. On the in-domain test set, the best combined model achieved an F1 score of 85.71, a statistically significant improvement over the 85.15 baseline.
  • Out-of-Domain Robustness: Performance gains were even more pronounced on out-of-domain data (increasing from 73.82 to 74.83 F1), indicating that syntactic variety aids generalization across different writing styles.
  • Recall Improvements: The most substantial gains were seen in recall, suggesting the model became more adept at identifying arguments in complex or unusual configurations.

Discussion: The Power of Syntactic Gymnastics

The core insight of this work is that models benefit less from simply more data and more from seeing the same semantic relationships expressed through different "syntactic gymnastics." By providing variants, the feature space for the classifier is effectively smoothed. Interestingly, automatically learned rules from Wikipedia outperformed sophisticated hand-crafted rules, as they captured a broader spectrum of natural linguistic shifts.

Conclusion

This research demonstrates that automatic text rewriting is a powerful, cost-effective method for boosting SRL performance. By teaching models to recognize semantic roles across diverse syntactic structures, the approach mitigates the bottleneck of human annotation. Future work may explore more sophisticated filtering techniques, such as n-gram language models for fluency, and the application of these methods to unsupervised semantic role induction or machine translation.

Summary updated

Abstract

Large-scale annotated corpora are a prerequisite to developing high-performance NLP systems. Such corpora are expensive to produce, limited in size, often demanding linguistic expertise. In this paper we use text rewriting as a means of increasing the amount of labeled data available for model training. Our method uses automatically extracted rewrite rules from comparable corpora and bitexts to generate multiple versions of sentences annotated with gold standard labels. We apply this idea to semantic role labeling and show that a model trained on rewritten data outperforms the state of the art on the CoNLL-2009 benchmark dataset.
BibTeX
@article{woodsend2014text,
  author = {Kristian Woodsend and Mirella Lapata},
  title = {{Text Rewriting Improves Semantic Role Labeling}},
  journal = {J. Artif. Intell. Res.},
  year = {2014},
  doi = {10.1613/jair.4431},
}
Figure