When we read the sentence, "The chef chopped the onions with a knife and sautéed them," our brains do not process the actions in isolation. We instantly recognize a web of constraints: the "chef" is the agent for both actions, the "onions" are the shared object, and a "knife" is a tool highly compatible with "chopping" but irrelevant to "sautéing."
Traditional natural language processing and computer vision systems, however, have long treated semantic analysis as a series of isolated classification tasks. In Semantic Role Labeling (SRL) or visual Situation Recognition (SR), models frequently predict the arguments of a verb (who, what, with what) independently, or analyze multiple verbs in a sentence as completely separate events.
This pointwise approach creates a fundamental blind spot. By ignoring how arguments co-occur and how multiple predicates interact, models frequently produce contextually inconsistent or contradictory predictions. To overcome this, a major research direction focuses on jointly modeling inter-argument and inter-predicate dependencies to capture the holistic semantic structure of sentences and scenes.
The First Dimension: Inter-Argument Interdependence
The first core insight of this research direction is that within a single predicate or event, semantic roles are highly mutually constrained. Knowing the identity of one argument provides critical context for identifying the others.
[ Pointwise Selectional Preference ]
Verb: "drink" ---> Subject: ? (Evaluated in isolation)
Verb: "drink" ---> Object: ? (Evaluated in isolation)
[ Joint Inter-Argument Modeling ]
Subject: "The car" <--- (Incompatible!) ---> Object: "gasoline"
\ /
\---> Verb: "drink" <---/
Breaking the Independence Assumption in Selectional Preferences
Early statistical models often treated grammatical case fillers as independent variables. However, Atsushi Fujii et al. demonstrated that ignoring argument co-occurrence severely limits a model's disambiguating power; by explicitly modeling the specific interactions between different case roles (such as subjects and objects), verb sense disambiguation accuracy improves significantly. Similarly, Kiyoaki Shirai et al. identified "inter-filler dependencies," showing that traditional dependency models fail when they evaluate the relationship between a head and its modifiers independently.
When models evaluate selectional preferences—the semantic constraints on which words fit together—doing so in a pointwise manner leads to errors. Kangqi Luo et al. argued that analyzing relation arguments in isolation prevents systems from fully disambiguating polysemous relation strings. To capture full sentence meaning, Deyi Xiong et al. emphasized the need for unified models that evaluate a verb and all of its arguments simultaneously, rather than treating subject and object preferences as independent. This mutual constraint is so powerful that, in unsupervised settings, Dmitry Ustalov et al. showed that simultaneously clustering predicates and their multiple arguments via triclustering captures semantic frames far more effectively than sequential or isolated grouping.
Interdependent Roles in Visual Situations
This problem is equally pronounced in computer vision, where "situation recognition" requires identifying an action and its associated semantic roles from an image. Arun Mallya observed that structured models struggle to capture these complex dependencies, often predicting contextually inconsistent entity combinations—such as an agent that cannot logically interact with a specific tool.
To resolve this, Ruiyu Li et al. and Ngai-Man Cheung et al. framed semantic roles as inherently interdependent. For instance, the identity of a "tool" role heavily depends on the "agent" and the "place." By allowing roles to query and condition on one another, models avoid disjointed or contradictory scene descriptions.
The Second Dimension: Inter-Predicate Interactions
The second dimension scales this intuition up from a single verb to the entire sentence. Natural language sentences rarely contain just one predicate; instead, multiple verbs and event-nouns typically overlap and compete for the same arguments.
[ Pointwise Multi-Predicate Analysis ]
Sentence: "The boy caught the ball and threw it."
- Step 1: Analyze "caught" ---> Agent: "The boy", Object: "the ball"
- Step 2: Analyze "threw" ---> Agent: ?, Object: ? (Processed from scratch)
[ Joint Multi-Predicate Analysis ]
Sentence: "The boy caught the ball and threw it."
\ / /
\ / (Shared Context)
v v v
[ caught ] <=======> [ threw ]
The "Isolated Island" Problem
Standard SRL systems often process each predicate in a sentence independently. As Haitong Yang et al. pointed out, because the vast majority of multi-predicate sentences involve shared arguments, relying on independent predicate analysis completely ignores these frequent structural overlaps. Treating multiple predicates as independent tasks fails to capture the holistic interactions between different semantic structures.
This limitation extends across sentences and domains. Svetlana Stoyanchev et al. noted that processing frame arguments independently prevents models from capturing complex inter-frame relationships. Furthermore, sequence labeling architectures frequently miss parallel items or subsequent arguments in sentences containing coordinating conjunctions, a flaw that Bing Liu et al. attributed to a failure to model the semantic links established by conjunctions.
Resolving Omitted Arguments (Zero Pronouns)
Modeling inter-predicate interactions is particularly vital for pro-drop languages like Japanese and Chinese, where subjects and objects are frequently omitted when they can be inferred from context.
- Shared Context: Hiroki Ouchi et al. demonstrated that analyzing how the semantic roles of one verb relate to those of another provides the shared context necessary to resolve ambiguous or omitted arguments.
- Distance-Free Modeling: Kentaro Inui showed that identifying omitted arguments becomes significantly more effective when a model considers the interactions between all predicates in a sentence simultaneously, leveraging the predicted structures of neighboring predicates to infer missing information in a target predicate.
- Clausal Context: Chen Chen 0004 et al. found that simple semantic compatibility checks fail if they only evaluate a verb and its potential subject in isolation; accurate zero pronoun resolution requires evaluating the entire, multi-argument clausal context.
- Event-Noun Integration: Hikaru Omori extended this joint modeling paradigm to include "event-nouns" (nouns that act like predicates, such as destruction or arrival), showing that verbs and event-nouns share argument context and must be resolved together to achieve high structural accuracy.
Algorithmic Mechanisms for Joint Inference
To move from pointwise prediction to joint modeling, researchers have developed several distinct architectural mechanisms that allow predicates and arguments to "negotiate" and satisfy global constraints.
1. Graph Neural Networks and Attention
To capture direct role-to-role relationships without forcing a rigid sequential order, Ruiyu Li et al. utilized Graph Neural Networks (GNNs) where semantic roles serve as nodes that pass messages to one another. For multi-predicate sentences, Kentaro Inui implemented pooling and attention mechanisms on top of deep recurrent networks, allowing distance-free interaction modeling across disparate verbs in a sentence.
2. Global Context Nodes and Capsule Networks
Instead of relying purely on local word representations, Xinchi Chen et al. introduced Capsule Networks featuring a global context node. This node acts as a high-level summary of the entire predicate-argument structure. Through an iterative routing process, the capsules negotiate to satisfy global structural constraints (e.g., ensuring a verb does not mistakenly receive multiple subjects), and can even infer missing arguments based on the roles already identified.
[ Capsule Routing with Global Context ]
Argument Capsules Global Context Node Structural Constraints
[ Agent Capsule ] --------\ /---> [ Max 1 Subject ]
[ Object Capsule ] ---------> [ High-Level Summary ] ------/----> [ Missing Argument Inference ]
[ Tool Capsule ] --------/
3. Global Optimization and Logic Frameworks
When hard logical constraints must be strictly enforced, researchers turn to global optimization techniques:
- Integer Linear Programming (ILP): In bilingual settings, Tao Zhuang et al. utilized ILP to enforce structural consistency across parallel texts, ensuring that predicate detection and argument labeling on both sides of a translation correct each other's errors simultaneously.
- Markov Logic Networks (MLNs): Iván V. Meza-Ruíz et al. combined first-order logic with probabilistic graphical models, allowing the system to jointly resolve predicate senses and their multiple arguments by maximizing a global probability score over the entire sentence structure.
4. Multi-Root Parsing Frameworks
Moving away from traditional sequence tagging entirely, the authors of Span-based Semantic Role Labeling as Lexicalized Constituency Tree Parsing reframed SRL as a lexicalized constituency tree parsing task. By treating predicates as roots and arguments as subtrees within a unified parsing framework, the model naturally accounts for the overlapping dependencies and constraints that exist between multiple predicate-argument structures in a single sentence.
Key Foundational Work
Semantics of Complex Sentences in Japanese (1994) — contributed 1 concept elements to this direction.
To what extent does case contribute to verb sense disambiguation? (1996) — contributed 1 concept elements to this direction.