Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks

Xinchi Chen, Chunchuan Lyu, Ivan Titov

Abstract: Xinchi Chen, Chunchuan Lyu, Ivan Titov. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Semantic Role Labeling (SRL) is the task of identifying "who did what to whom" in a sentence. While modern systems are highly effective at local word-level encoding using LSTMs, they often struggle with global consistency. For example, a system might erroneously assign the "Subject" role to two different words for the same verb, even though most verbs only allow one. Traditional systems attempted to fix this with rigid "hard constraints," which are often too brittle for natural language. This paper introduces CapNetSRL, which uses Capsule Networks to allow different parts of a sentence to "negotiate" role assignments through a data-driven, iterative process.

The CapNetSRL Architecture

The core innovation of CapNetSRL is the application of Capsule Network logic to sentence structure.

  1. Word Capsules: Instead of a single vector, the model creates "word capsules" for every possible semantic role, encoding how well a word fits a specific role (e.g., is "injury" a likely "signer"?).
  2. Role Capsules & Dynamic Routing: Through multiple iterations of Dynamic Routing, the model refines connections between Word Capsules and "Role Capsules" (representing the overall proposition). If a word is a strong candidate for a role, its connection is strengthened.
  3. The Global Node: To capture high-level dependencies, the authors added a "Global Node." This node summarizes the entire proposition, allowing the model to recognize relationships between different roles (e.g., if a "Buyer" is found, the model should also look for a "Seller").
The architecture of CapNetSRL, illustrating the flow from Bi-LSTM layers to Word and Role Capsules, integrated with a Global Node for proposition-level context.

Iterative Refinement and Soft Constraints

A significant insight from this research is that explicitly forcing uniqueness via hard rules can actually hurt performance. Instead, the Dynamic Routing mechanism naturally learns "soft constraints." As the model iterates, it becomes "aware" of the global structure; if one word is identified as a strong subject, the routing mechanism suppresses the likelihood of other words taking that same role.

The number of propositions with duplicate arguments decreases significantly over iterations in the full CapsuleNet model, demonstrating how dynamic routing enforces structural constraints.

The authors found that the Global Node is essential for this refinement. Without it, the process acts primarily as a filter (increasing precision). With it, the model can actually "discover" missing arguments, significantly improving recall.

Key Results

CapNetSRL was evaluated on the CoNLL-2009 shared task datasets across seven languages, achieving state-of-the-art (SOTA) results on five of them (English, Chinese, Japanese, Czech, and Spanish).

  • English Performance: The model reached an F1 score of 91.06% on the English test set, outperforming previous strong baselines.
  • Robustness to Domain Shift: On out-of-domain data, the model showed a significant jump to 82.72% F1, suggesting that modeling structural interactions helps the model generalize better than memorizing word patterns.
  • Efficiency: The iterative process is highly efficient, with peak performance typically reached in just 2 to 3 iterations.

Conclusion

By treating semantic roles as capsules and using dynamic routing, CapNetSRL effectively captures global structural constraints without relying on hand-coded rules. This approach proves particularly robust for out-of-domain text. Future work will explore adapting this architecture to span-based SRL and modeling interactions between multiple overlapping predicates simultaneously.

Summary updated

BibTeX
@inproceedings{chen2019capturing,
  author = {Xinchi Chen and Chunchuan Lyu and Ivan Titov},
  title = {{Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=RgpBR441OX-},
}
Figure