Phrase grounders trained only on image–caption pairs learn which image region a phrase like "the horse" refers to, but they read captions as nouns joined by surface verbs. When a caption describes an event with a noun ("a woman's leap over the hurdle") instead of a verb ("a woman leaps over the hurdle"), the grounder's text parser tends to treat "leap" as a physical object. It then loses the fact that the woman is the one doing something and the hurdle is what it is done to. Language research has fixed a closely related failure on the text side. Parsers that label "who did what to whom" (semantic role labeling) and are trained on verbs collapse on nouns, dropping from 84.7 to 16.4 F1. Mapping nouns and verbs into one shared inventory of about 425 coarse event types (called frames) largely repairs this.
This proposal uses that shared inventory as the organizing key for a spatial prior that is learned without boxes. For each frame and each pair of roles (for example, the agent and the thing ridden in RIDE), the model learns how the two boxes typically sit relative to each other, pooling evidence from every caption that expresses that frame, whether through a verb or a noun. During weakly supervised training, this prior helps choose among candidate boxes and pushes box refinement toward whole objects. Our bet has two parts: grounding accuracy should stop depending on whether an event was worded as a verb or a noun, and frame-level role geometry should reduce the familiar failure where a weakly supervised model boxes only the most distinctive part of an object.
Core Claim
Weakly supervised grounders can be made invariant to the part of speech of event words, and more complete in the boxes they draw, by indexing a learned relative-geometry prior over role pairs by coarse, category-independent frames rather than by surface verbs or dependency arcs. The new piece is not "use a better text parser." It is that the frame abstraction serves as the unit of statistical sharing for spatial evidence. Captions with verbs and captions with nouns then train the same geometry model, and that model feeds back into box selection. A text-only fix, such as rewriting nouns as verbs, cannot provide this sharing across different words for the same event, and it provides no geometry.
Where This Sits
Three lines of prior work motivate the gap. On the language side, Exploring Non-Verbal Predicates in Semantic Role Labeling shows the verb-to-noun collapse and finds that the coarse VerbAtlas frames cut predicate disambiguation error by 38% compared with fine PropBank senses. NounAtlas maps 10,086 nominal predicates onto those frames and trains a joint noun-plus-verb role labeler that reaches 72.4 F1 on nominal predicates without losing verbal accuracy. We reuse that parser as a component; it supplies the method. QANom reports that 43% of arguments of nominalizations go unexpressed, against 29% for verbs. This constrains us: noun-worded captions will more often mention only one role, so the prior must tolerate a missing partner.
On the vision side, the closest system is Relation-aware Instance Refinement for Weakly Supervised Visual Grounding. It builds a graph over object proposals whose edges come from a dependency parse and refines boxes with a self-taught regression loss, which treats its own confident boxes as pseudo-labels. It reaches 59.27% top-1 accuracy on Flickr30k Entities and 37.68% on ReferItGame. We keep its proposal graph and refinement loop. We replace dependency-arc edges with frame-role edges and add the geometry prior. Its parser is exactly the component that misreads "inspection of the engine" as an object.
Visual Semantic Role Labeling supplies the idea of a spatial displacement prior between agent and role object. It hand-specifies a Gaussian per verb class, trains it with full boxes, and covers only 26 verbs. We learn the prior per frame, with no boxes, across nouns and verbs.
Two structured-alignment works show that predicate structure helps weakly supervised vision, and both stop short of box geometry. Weakly Supervised Visual Semantic Parsing treats predicates as graph nodes matched to image-level triplets. Cross-media Structured Common Space for Multimedia Event Extraction aligns meaning-representation graphs of text with situation graphs of images, but at the event-classification level. Neither models nominal wording, and neither refines box extent.
Finally, ExpAlign closes much of the localization gap for open-vocabulary grounding by treating token-to-region alignment as multiple-instance learning, meaning each phrase is known to match some region in the bag but not which. It notes that its frozen CLIP text encoder behaves like a bag of words and misses relational structure. That failure is the one frame-role edges target.
One tension needs addressing directly. Orlando et al. find that coarse abstractions help language generalization, while ExpAlign finds that coarse text representations hurt localization. We resolve this by separating the two jobs. Tokens and phrase spans still decide which region a phrase matches. The frame is used only to decide which geometry statistics apply. Whether this separation holds up is itself an ablation below.
Research Program
The pipeline has three parts. First, every training caption is parsed with the NounAtlas noun-plus-verb role labeler. This yields frame instances whose predicate may be a verb or a noun, each with role-labeled argument spans such as Agent, Theme, or Instrument that link to the caption's entity phrases. Second, the proposal graph from Liu et al. is built over region candidates. Each frame instance adds edges between the proposal sets of its role phrases, labeled by frame and role pair. Third, the latent box assignment is chosen by a score that adds a frame-indexed geometry term to the usual phrase–region match:
$$ S(b_a, b_r) = \phi(t_a, b_a) + \phi(t_r, b_r) + \lambda \log p_f\big(g(b_a, b_r)\big), $$
Here $b_a$ and $b_r$ are candidate boxes for the two role phrases $t_a$ and $t_r$, and $\phi$ is the existing phrase–region match score. The function $g$ gives the pair's relative geometry: log scale ratio, normalized center offset, and overlap. $p_f$ is a mixture density over that geometry for frame $f$, and $\lambda$ weights the prior.
Training alternates, in expectation–maximization style. Confident assignments, those with a high score margin, re-estimate $p_f$. The updated $p_f$ then rescores assignments and chooses the pseudo-boxes used by the self-taught regression loss. Frames with few confident examples back off to a pooled prior over their role pair. When a noun-worded caption expresses only one role, the geometry term drops out, and the instance still contributes to $\phi$.
The mechanism against part-only boxes is indirect but testable. A head-only box for the agent of RIDE produces a scale ratio and offset relative to the horse that fall in the tails of $p_f$. The prior therefore favors the full-person proposal and supplies a full-person pseudo-target for regression.
Scope is limited to deverbal and event nouns covered by NounAtlas. Adjectival predicates and implicit-argument grounding are left out, because the file shows no resource that covers them adequately.
Baselines
The naive reference point is the dependency-parse grounder of Relation-aware Instance Refinement run unchanged. It reports 59.27% top-1 on Flickr30k Entities and 37.68% on ReferItGame, and it is also the strongest reported weakly supervised grounder in the file on these benchmarks. We will retrain it ourselves with the same proposals, backbone, caption data, and training schedule. We will confirm that we reproduce roughly its reported number before comparing anything, so that differences reflect edges and priors rather than the training pipeline. Beating it means a statistically reliable gain over the reproduced number on the full test set, plus a larger gain on the noun-event subsets described below.
The strongest text-side competitor is LLM rewriting followed by the same grounder. An LLM rewrites each noun-worded event clause into verbal form. The original entity spans are kept by string alignment, and the Liu et al. grounder runs on the result. This baseline gets the same tuning budget as our system, including prompt selection on the development split. Beating it requires gains on the paraphrase-controlled subset and on lexically diverse frames, not merely matching it on rewritten verbs.
The strongest vision-side competitor pairs a CLIP-based token grounder in the style of ExpAlign with a class-agnostic box adjuster in the style of LBBA. ExpAlign reports 36.2 rare-class AP on LVIS minival. We found no reported result on Lacuna for ExpAlign on Flickr30k Entities, so we will run it on our splits if code is available. If it is not, we will run a CLIP-feature multiple-instance grounder with the same proposals. Because LBBA uses fully boxed auxiliary data, we will also give our model an LBBA adjuster in one run. That run tests whether the frame prior still adds something once box quality is improved from outside.
Three ablations isolate the key ingredient. The first swaps the NounAtlas parser for a verb-only role labeler, which isolates part-of-speech unification. The second removes $p_f$ while keeping frame-role edges, which isolates the geometry prior. The third indexes $p_f$ by fine PropBank senses instead of VerbAtlas frames, which tests the coarse-versus-fine tension. A fourth run keeps $p_f$ but indexes it by the surface lemma. This separates the sharing across different words that frames provide from generic relational priors.
Why not just…?
Why not just rewrite nouns into verbs with an LLM and use an existing grounder? This is the real threat. On paraphrase-controlled captions it may recover most of the verb–noun gap on the text side. Two things it cannot do motivate the proposal. It still leaves "rides," "mounting," and "equestrian jump" as different lemmas with separate statistics. It also adds no geometric constraint, so it cannot affect part-only boxes. NounAtlas found LLM-generated nominalizations useful only after a separate role-projection step and explicit nominal training, which suggests role binding does not survive rewriting for free. We will settle it empirically. If rewrite-plus-Liu matches our full model on the paraphrase subset and our lemma-indexed ablation matches our frame-indexed model, then the part-of-speech story is moot. In that case we would report the contribution as the geometry prior alone.
Why not just use CLIP features with a class-agnostic box adjuster? LBBA raises Pascal VOC 2007 detection from 41.2% to 56.6% mAP, so better boxes from outside do help. But the adjuster refines a box that has already been chosen. It does not decide which person is the agent. Its authors' own text encoder, per ExpAlign, lacks relational awareness. Our LBBA-augmented run tests whether the two effects add or overlap.
Why not just query a grounder with QANom-style questions? Questions like "what was inspected?" do recover the role meaning. However, each question grounds one phrase on its own, with no joint constraint between agent and patient boxes. The CCA phrase-to-region baseline in Flickr30k Entities, at 11.22% mAP, shows how weak independent phrase matching is compared with relational refinement. We will run question-augmented queries as an extra baseline if time permits. We expect them to help recall of the frame word but not box completeness.
Proposed Experiments
Three evaluation sets are used. The first is the standard Flickr30k Entities and ReferItGame test sets, measured by top-1 accuracy at IoU of at least 0.5. The second is a noun-event subset of Flickr30k Entities: phrases that the joint parser marks as arguments of a nominal predicate, checked by hand. We will report its size, since coverage is unknown and could be small. The third is a paraphrase-controlled set built from Flickr30k test captions whose events are verbal. Each caption is nominalized, the rewrite is verified by humans, and the gold entity boxes are kept. Every image–phrase pair then appears in both wordings.
For role localization, we train on COCO captions without boxes and evaluate agent and role detection on V-COCO. The fully supervised reference there is 26.4% role mAP from Visual Semantic Role Labeling; we do not expect to match it without boxes.
The first decisive check is the verb–noun accuracy gap on the paraphrase-controlled set. The reproduced Liu et al. grounder should show a clear drop on the noun wording. Our model should shrink that gap substantially without losing accuracy on the verb wording. The second decisive check targets part-only boxes. Among agent predictions on person phrases, we measure the fraction that lie mostly inside the gold box but reach IoU below 0.5. This is the failure behind MELM's 2.0% person AP in Min-Entropy Latent Model. Removing $p_f$ should raise that fraction.
The third check is the index comparison. VerbAtlas-indexed priors should beat PropBank-indexed and lemma-indexed ones, most of all on rare frames. If fine senses win instead, that would favor ExpAlign's side of the coarse-versus-fine tension over Orlando et al.'s, and it would argue for indexing by lemma or sense rather than by frame.
References
- Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities
- NounAtlas: Filling the Gap in Nominal Semantic Role Labeling
- QANom: Question-Answer driven SRL for Nominalizations
- Relation-aware Instance Refinement for Weakly Supervised Visual Grounding
- Visual Semantic Role Labeling
- Weakly Supervised Visual Semantic Parsing
- Cross-media Structured Common Space for Multimedia Event Extraction
- ExpAlign: Expectation-Guided Vision–Language Alignment for Open-Vocabulary Grounding
- Boosting Weakly Supervised Object Detection via Learning Bounding Box Adjusters
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- Min-Entropy Latent Model for Weakly Supervised Object Detection