Abstract
Most supervised language processing systems show a significant drop-off in performance when they are tested on text that comes from a domain significantly different from the domain of the training data. Semantic role labeling techniques are typically trained on newswire text, and in tests their performance on fiction is as much as 19% worse than their performance on newswire text. We investigate techniques for building open-domain semantic role labeling systems that approach the ideal of a train-once, use-anywhere system. We leverage recently-developed techniques for learning representations of text using latent-variable language models, and extend these techniques to ones that provide the kinds of features that are useful for semantic role labeling. In experiments, our novel system reduces error by 16% relative to the previous state of the art on out-of-domain text.
Introduction
Semantic Role Labeling (SRL) is a fundamental task in Natural Language Processing (NLP) that involves identifying the predicate-argument structure of a sentence—essentially determining "who did what to whom." While supervised SRL systems have achieved high performance on in-domain data, they often suffer from a dramatic "performance cliff" when applied to out-of-domain text. For instance, a model trained on the Wall Street Journal (WSJ) newswire corpus typically sees its F1 score plummet by nearly 20% when tested on the Brown Corpus of fiction. This fragility stems from a reliance on brittle, lexicalized features that do not generalize across different writing styles and vocabularies.
The Span-HMM Approach
To address the problem of domain shift, we propose moving away from sparse, surface-level features toward unsupervised, latent-variable representations. The core innovation of this work is the Span-HMM.
In traditional SRL models, the "path" between a predicate and an argument is represented as a sequence of words or tags. These paths are highly specific and rarely repeat in new domains, leading to severe feature sparsity. The Span-HMM addresses this by introducing a single "span node" that is responsible for generating the entire sequence of words between a predicate and its argument.
By training this model on large amounts of unlabeled text, we can represent any word sequence as a single latent categorical state. This "condenses" millions of possible word combinations into a small set of features that are much more likely to be encountered during training, even if the specific words in the target domain are novel. We further enhance robustness by combining multiple independent versions of these models into a Multi-Span-HMM (MSH) using a Conditional Random Field (CRF) framework.
Experimental Results
We evaluated our system by training on the CoNLL 2005 WSJ data and testing on both the WSJ (in-domain) and the Brown Corpus (out-of-domain).
Our Multi-Span-HMM achieved a state-of-the-art F1 score of 73.8 on the Brown corpus, representing a 16% relative error reduction compared to previous benchmarks. Crucially, our approach significantly narrowed the performance gap between domains. While previous systems experienced a drop of approximately 12 points when moving from news to fiction, the Multi-Span-HMM saw only a 5.4-point decrease.
Beyond aggregate scores, the model showed particular strength in identifying "unseen" predicates and maintaining accuracy for long-distance arguments, where the complexity of intervening text typically causes traditional models to fail.
Conclusion
The success of the Span-HMM suggests that the primary obstacle to open-domain SRL is not the learning algorithm itself, but the specificity of the features used. By leveraging latent-variable models to learn domain-invariant representations of word spans, we can build SRL systems that are far more robust to the stylistic and lexical variations found in diverse text sources. This work brings us closer to the goal of "train-once, use-anywhere" language processing systems.
Summary updated