Models that read a sentence and extract "who did what to whom" have improved by letting predictions talk to each other. When a sentence contains several verbs that share or compete for the same phrases, a model that lets the candidate roles of one verb inform the others makes fewer inconsistent guesses. This proposal asks what that same coupling does to the model's ability to know when it is wrong. We conjecture that coupling helps accuracy but hurts calibrated abstention. When roles condition on each other, a wrong guess can be propped up by its neighbors, so the model ends up confidently and jointly wrong. If that is right, the confidence scores of these models are a poor guide for deciding which parts of a parse to trust.
We propose a way to abstain on individual parts of a parse rather than on the whole sentence. For each predicted role, the model makes two predictions: one with the context from neighboring roles and predicates, and one with that context removed. It then compares them. A role that survives without the context is supported by direct evidence. A role that exists only because of the context is being inferred from dataset regularities, and it is a candidate for abstention. We pair this signal with a risk bound computed over whole sentences rather than individual arcs, because errors within a sentence are correlated. We then test the idea on span and dependency semantic role labeling and on visual situation recognition.
Background and the Gap
Semantic role labeling (SRL) assigns each phrase a role relative to a predicate, such as agent, patient, or instrument. Two lines of work model interactions among these assignments, and they supply the base predictors here. Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks uses dynamic routing, an iterative procedure in which candidate words repeatedly negotiate which role slot they belong to, together with a sentence-level summary node. It suppresses duplicate arguments within two to three routing iterations and reports 91.06 F1 on English CoNLL-2009 and 82.72 out of domain. Span-based Semantic Role Labeling as Lexicalized Constituency Tree Parsing enforces non-overlapping argument spans exactly, by decoding each predicate's arguments as a tree with dynamic programming. It reports 87.99 F1 on CoNLL-2012 and 83.07 on the out-of-domain Brown corpus.
In vision, Attention-Based Context Aware Reasoning for Situation Recognition lets each role's query attend to neighboring roles before looking at the image. This raises full-frame accuracy on imSitu from 14.28% to 18.58%. The same paper also found that stacking a second context module lowered performance, which it attributed to over-reliance on dataset priors. That negative result is the motivation for our central hypothesis. None of these papers evaluates whether its confidence scores are useful for abstention.
Selective prediction means letting a model decline to answer so that errors on the answers it does give stay low. Its standard tools assume one decision per input. Selective Classification for Deep Neural Networks supplies our method for guarantees. It sets a threshold on validation data so that test error stays below a target with high probability, and it shows that maximum softmax probability is a hard baseline to beat. Its binomial bound treats decisions as independent, which arcs in the same sentence are not. Towards Improving Selective Prediction Ability of NLP Systems trains a secondary calibrator on surface features and training dynamics. It shows that maximum probability is overconfident under domain shift, which suggests the effect we look for should be largest on Brown and on the out-of-domain CoNLL-2009 split.
Three papers are the closest structured neighbors, and each leaves the gap open. EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records abstains on a whole SQL program at once using entropy thresholds. Improving Cascade Routing for Structured Attribute Generation with Heterogeneous Confidence shows that confidence behaves differently across output types, and that type-specific thresholds reach 0.8146 precision while deferring 19.3% of traffic, against 26.4% for a global threshold. It treats every slot independently. Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning breaks a question into simpler sub-questions and uses the answers to rescue low-confidence correct predictions. That is also a form of evidence-based confidence, but it operates on single answers, not on interacting role structures. Our proposal differs from all three in two ways. The unit of abstention is a part of a coupled structure, and the confidence signal is specifically about whether the coupling itself manufactured the prediction.
Core Claim
We make two claims, each of which can be refuted. The first is diagnostic. As interaction strength increases, measured by routing iterations in the capsule model, tree marginals versus local arc scores in the span parser, and context attention on versus off in the situation model, arc-level accuracy rises. Over the same range, the area under the arc-level risk–coverage curve falls less than accuracy alone would predict, or even rises. The reason is that errors become more confident and more correlated within a sentence. We expect this gap to be larger out of domain.
The second claim is constructive. Let $s_c(a)$ be the probability the model gives arc $a$ with interaction context and $s_0(a)$ the probability for the same arc with context severed. Both come from one network trained with random context dropout, so it can run in either mode. A selection score that combines $s_c(a)$ with the disagreement $s_c(a) - s_0(a)$ should identify errors the joint probability misses. It should give a better risk–coverage trade-off at the arc level than any single-mode confidence, at the cost of one extra forward pass.
Research Program
The work has three parts.
The first part builds dual-mode versions of the three base models. For the capsule model, the severed mode takes predictions before any routing update. For the span parser, the severed mode uses the local biaffine arc scores without tree decoding, and the coupled mode uses tree marginals computed by the inside–outside algorithm over the same chart. For the situation model, the context-attention step is skipped. Each model is trained with context dropped on a random fraction of steps. We check that its coupled-mode accuracy matches the published numbers above before any selective experiment.
The second part defines abstention over sub-structures. The system outputs a retained sub-graph of predicate–argument arcs. When one phrase is contested by roles under two different predicates, the natural unit is the whole contention set: abstaining on one assignment while keeping its rival would silently commit to the rival. We therefore compare arc-wise selection with contention-set selection, where the set's score is its least trustworthy member.
The third part supplies the guarantee. Arcs in a sentence share errors, so a per-arc binomial bound would be too optimistic. We instead treat sentences as the independent units. Each sentence's loss is its fraction of retained arcs that are wrong, a number in [0, 1]. We choose the threshold $\theta$ so that an upper confidence bound on the mean of this loss over validation sentences stays below the target risk $r^*$ with probability $1-\delta$. This adapts the threshold-selection logic of Geifman and El-Yaniv from independent instances to independent sentences.
Baselines
The naive reference points are full-sentence abstention and no abstention. Full-sentence abstention scores each sentence by its lowest-confidence arc or by mean entropy, in the manner of EHRSQL, and keeps all arcs or none. It shows how much is gained simply by abstaining on parts. It is beaten if arc-level selection retains more correct arcs at every matched risk level.
The strongest existing competitor is per-arc confidence from the coupled model with a separate threshold for each role type. Concretely, this means tree marginals for the span parser and final-iteration routing probabilities for the capsule model, following the type-aware thresholding of the cascade-routing paper. It is the strongest because it already uses the structured model's best probability estimate and already corrects for heterogeneity across role types. The cascade paper's 19.3% versus 26.4% deferral result is the relevant precedent, although no reported result found on Lacuna gives arc-level risk–coverage numbers for SRL. Our method gets the same per-role-type thresholds, the same validation split, and the same threshold-search procedure, so the only difference is the score. Beating it means a lower arc-level area under the risk–coverage curve on all three benchmarks, a bootstrap interval over test sentences that excludes zero, and the largest margin on Brown and the CoNLL-2009 out-of-domain split.
We also run the secondary-calibrator approach of Varshney et al., adapted to arcs. It is a random forest over arc probability, span length, distance to the predicate, and the number of predicates in the sentence, and we give it the same validation data. This tests whether a learned combination of cheap features already captures what the disagreement signal captures.
Two ablations isolate the key ingredient. One replaces the severed-mode prediction with a second independently seeded coupled model, keeping the rest identical. This separates "disagreement between context and no context" from generic two-model disagreement. The other drops contention sets and selects arc by arc, which isolates the grouping.
Why not just…?
Why not just threshold the structured model's own arc marginals per role type? This is the real threat. Tree marginals already account for competition among spans, so they may already reflect the uncertainty we attribute to context. We do not argue around this. The diagnostic in claim one tests it directly. We bin arcs by marginal probability and ask whether the disagreement signal still separates correct from incorrect arcs within bins. If it does not, the disagreement signal adds nothing, and the contribution reduces to the sentence-level guarantee and the contention-set unit, which we would report as the result.
Why not just use MC-dropout or a small ensemble to get arc uncertainty? Geifman and El-Yaniv found that MC-dropout did not reliably beat softmax response. Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs notes that multi-sample methods multiply inference cost. More to the point, sampling noise measures parameter uncertainty, not whether a prediction rests on image or text evidence versus role priors. The two-seed ablation, run at equal cost to our two-pass method, measures this directly.
Why not just enforce hard consistency constraints and abstain when they cannot be satisfied? The capsule paper reports that rigid constraints hurt SRL relative to learned soft negotiation. Constraint violations are also rare in models that already decode with trees, so the rule would almost never fire. Violation-triggered abstention is therefore a weak signal, and we include it only as a sanity baseline.
Proposed Experiments
The first experiment tests claim one. We plot arc accuracy and arc-level area under the risk–coverage curve against interaction strength for all three models, in domain and out of domain, alongside expected calibration error computed separately for arcs that change between severed and coupled modes. The decisive pattern is that arcs which appear only in coupled mode are wrong more often than their confidence implies. If that pattern is absent, the hypothesis fails.
The second experiment tests claim two. It compares risk–coverage curves for all baselines on CoNLL-2009 (in and out of domain), CoNLL-2012 with Brown, and imSitu. On imSitu, a retained arc is a role–value pair, and it counts as correct if it matches any of the three reference frames. The third experiment checks the guarantee. At target risks of 2%, 5%, and 10% and $\delta = 0.05$, over 100 random calibration/test splits, we report the realized violation rate and the coverage achieved. We contrast this with a naive per-arc binomial bound, which we expect to violate its target more often than $\delta$ because within-sentence errors are correlated.
The fourth experiment reports how many sentences retain a complete frame for at least one predicate under each method. This measures practical value: a partial parse is useful downstream only if it contains whole propositions, not scattered arcs.
References
- Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks
- Span-based Semantic Role Labeling as Lexicalized Constituency Tree Parsing
- Attention-Based Context Aware Reasoning for Situation Recognition
- Selective Classification for Deep Neural Networks
- Towards Improving Selective Prediction Ability of NLP Systems
- EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records
- Improving Cascade Routing for Structured Attribute Generation with Heterogeneous Confidence
- Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
- Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs