Abstract
The current state-of-the-art end-to-end semantic role labeling (SRL) model is a deep neural network architecture with no explicit linguistic features. However, prior work has shown that gold syntax trees can dramatically improve SRL, suggesting that neural network models could see great improvements from explicit modeling of syntax. In this work, we present linguistically-informed self-attention (LISA): a new neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-of-speech, predicate detection and SRL. For example, syntax is incorporated by training one of the attention heads to attend to syntactic parents for each token. Our model can predict all of the above tasks, but it is also trained such that if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model. In experiments on the CoNLL-2005 SRL dataset LISA achieves an increase of 2.5 F1 absolute over the previous state-of-the-art on newswire with predicted predicates and more than 2.0 F1 on out-of-domain data. On ConLL-2012 English SRL we also show an improvement of more than 3.0 F1, a 13% reduction in error.
Introduction
Semantic Role Labeling (SRL) is a fundamental task in natural language processing that involves identifying the predicate-argument structure of a sentence—essentially answering "who did what to whom." Historically, SRL systems were heavily dependent on hand-crafted syntactic features and complex pipelines. The advent of deep learning led to the rise of "syntax-free" models, primarily based on LSTMs, which outperformed traditional systems by learning representations directly from raw text.
Despite their success, these neural models often fail on errors that are fundamentally syntactic, such as the incorrect attachment of prepositional phrases. Previous attempts to re-integrate syntax into neural architectures often introduced significant complexity or computational overhead. This paper introduces LISA (Linguistically-Informed Self-Attention), a model that integrates syntax into the modern Transformer architecture efficiently through multi-task learning and supervised attention.
The LISA Model
LISA builds upon the Transformer's multi-head self-attention mechanism. Unlike standard Transformers where attention heads learn patterns autonomously, LISA explicitly supervises the model to learn linguistic structure.
Syntactically-Informed Self-Attention
The core innovation is the supervision of a specific attention head to attend to the syntactic head (parent) of each token. This effectively embeds dependency parsing as an auxiliary task within the self-attention mechanism. By forcing the model to "pay attention" to syntactic parents, the resulting representations are naturally imbued with structural information.
Multi-Task Learning (MTL)
LISA is trained to simultaneously perform four distinct tasks:
- Part-of-Speech (POS) Tagging: Identifying word categories.
- Predicate Detection: Locating the verbs or nouns that anchor the semantic roles.
- Dependency Parsing: Predicting the syntactic structure via the supervised attention head.
- Semantic Role Labeling: The primary task, utilizing a bilinear scoring operation between predicate and role representations.
One unique advantage of this architecture is Parse Injection. Because syntax is handled via an attention head, a higher-quality syntactic parse from an external tool can be injected at test time by simply overriding the attention weights of the "syntax head." This allows the model to benefit from external expertise without requiring re-training.
Experimental Results
The model was evaluated on the CoNLL-2005 (WSJ and Brown) and CoNLL-2012 (OntoNotes) benchmarks. LISA demonstrated significant improvements over previous state-of-the-art models, particularly when using ELMo contextualized representations.
As shown in the results, LISA achieved a substantial increase of 2.5 F1 on the WSJ newswire test set. Even more impressive was the performance on the out-of-domain Brown corpus, where it achieved an improvement of over 3.5 F1. This suggests that while vocabulary and style vary across domains, the underlying rules of syntax remain stable, allowing a syntactically-aware model to generalize much more effectively.
Discussion and Conclusion
LISA demonstrates that supervising the "black box" of self-attention with linguistic knowledge provides a powerful scaffold for semantic tasks. Qualitative analysis indicates that this syntactic awareness specifically helps the model identify argument span boundaries, a common failure point for syntax-free models.
Furthermore, LISA is computationally efficient because it encodes the sentence only once for all predicates, unlike models that require separate passes for each predicate. The ability to modularly inject external parses at inference time provides a practical path for future-proofing SRL systems as dependency parsing technology continues to advance. In summary, LISA proves that explicit linguistic structure remains a vital component for achieving robust and generalizable performance in deep learning models for NLP.
Summary updated