Open-vocabulary segmenters are models that outline, pixel by pixel, whatever a user names in free text. They work well for concrete nouns ("car", "shed"). They break when the noun carries an abstract or diffuse modifier: "the abandoned car", "the dilapidated shed", "the frightened child". The modifier has no outline of its own. Encoding the whole phrase as one text vector lets it pull the prediction toward the wrong category or toward arbitrary regions. Linguistics already describes the right division of labour. In a noun phrase, the head noun fixes what kind of thing is meant, and the modifier narrows which instances qualify.
We propose to build that asymmetry into segmentation. The head noun alone decides where the candidate masks are. The modifier never draws pixels. It only decides which of those masks are kept. A small scorer makes that decision. It sees each candidate object together with its surrounding scene, and its reading of the modifier changes with the head noun, so "old" means something different for a car than for a man. The claim is sharper than "decompose the phrase". For modifiers that have no spatial footprint of their own, routing the modifier into mask formation is harmful. Routing it only into instance selection, with head-dependent meaning and scene context, recovers most of the lost accuracy. The advantage should grow as modifiers become more abstract.
Core Claim and Motivation
Three bodies of evidence motivate the design.
First, head dominance is robust in text. The head noun determines which relations a compound can express (Stanovsky et al.). Language models generalize better when the head is held fixed and the modifier varies than the reverse (Rambelli et al.). In human ratings, the head's meaning tracks the whole compound's meaning more closely than the modifier's does (Ramisch et al.).
Second, abstract and non-visual attributes do not map to coherent image regions. They are spread over several regions or implied by how objects are arranged (Hu et al.). Attention-based zero-shot recognizers fail on behavioural and non-visual attributes (ZSLViT).
Third, current segmenters encode the phrase as a whole. The segmenter trained on the largest data engine, SAM 3, which segments every instance matching a short noun phrase, states that it cannot handle multi-attribute queries. LSeg, which scores pixels against CLIP text embeddings, forces pixels onto the nearest listed label when the true label is absent.
Put together, the obvious failure is what we call head drift. The modifier's text signal moves the phrase embedding away from the head category, so masks land on the wrong kind of object. The prediction is that a head-anchored design removes head drift by construction. What remains is a pure instance-selection problem. Instance selection is the only part of the task where the abstract modifier genuinely matters.
A second strand of evidence shapes how the selection step is modelled. Two findings bear on the text side. In language models, the embedding of a compound is predicted well by a linear map of its parts: ridge regression reaches about 0.92 cosine similarity (How do Transformer Embeddings Represent Compositions?). CLIP's text encoder alone keeps attribute–object binding, with linear probes reaching up to 99%, and the binding is lost only when text is matched against images (Koishigarina et al.). One finding bears on the visual side. The same attribute looks different on different objects (EVA), which argues against any single fixed modifier direction. We therefore model the modifier as a shift in text space whose direction depends on the head noun: linear in the modifier, but chosen per head.
We state the scope limit up front. For lexicalized and figurative compounds, the modifier rather than the head decides whether the phrase is read literally (Müller et al.; McCarthy et al.). A "dark horse" is not a horse. Our method would fail on such phrases by design, so we treat them as an explicit out-of-scope stress test rather than hiding them.
Research Program
The system has three parts.
The first part is a parser. A dependency parser extracts the head noun and its modifiers from the query. MCRES uses constituency parses for a similar purpose, to build meta-learning curricula for referring segmentation. Here the parse instead controls which part of the phrase reaches which module.
The second part is a head-only proposer. A frozen open-vocabulary segmenter, SAM 3 by default, receives only the head noun and returns every instance of that category. SAM 3's presence head, which first predicts whether a concept appears in the image at all, is reused unchanged.
The third part is the modifier scorer. Each candidate instance $i$ is described by two feature vectors taken from the segmenter's own image features: $f_i$, pooled inside the mask, and $c_i$, pooled over a ring of surrounding context. The scorer computes
$$s_i = \sigma\big(\langle P[f_i; c_i],\; t_m + A(t_h)\,t_m \rangle\big),$$
where the symbols are as follows. $t_h$ and $t_m$ are the frozen CLIP text embeddings of the head and the modifier. $A(t_h)$ is a low-rank matrix produced by a small network from the head embedding, and it changes how the modifier is read for that head. $P$ projects the concatenated visual features into text space. $\sigma$ is the logistic function.
An instance is kept if $s_i$ exceeds a threshold. The threshold is calibrated once on a validation split. A multi-modifier query multiplies the scores of its modifiers.
Training needs no masks, only object-level attribute labels. VAW supplies boxed object instances with explicit positive and negative attribute labels. We turn each box into a mask by prompting SAM with it and train $P$ and $A$ with a binary loss. MIT-States and C-GQA, image-level state–object datasets used by EVA, provide additional pairs for pretraining. Everything else stays frozen, so the concrete-noun behaviour of the segmenter cannot degrade.
The analysis side gives the proposal its scientific content. We collect human concreteness ratings for the modifier inventory from annotators. We then measure how every method's gap to a head-plus-oracle-selection upper bound varies with modifier concreteness. We also measure head drift directly: the fraction of predicted masks whose instance falls outside the head category.
The hypothesis is two-sided. For concrete modifiers such as colour or texture, holistic phrase encoding should roughly match the factorized design, since these modifiers do have pixels. For abstract modifiers, holistic encoding should lose accuracy mainly through head drift, and the factorized design should not. If the gap does not open with abstractness, the core claim is wrong.
Baselines
Oracle-free floor. SAM 3 is prompted with the head noun alone and all head instances are returned. This is the "ignore the modifier" floor. It gets head localization right and selection wrong, so any method that uses the modifier must beat it on precision without giving up recall.
Strongest holistic segmenter. SAM 3 is prompted with the full phrase. It is the strongest existing open-phrase segmenter: the SAM 3 paper reports 48.8 zero-shot mask AP on LVIS against a previous best of 38.5. It is run zero-shot, and also after fine-tuning its text-conditioning path on the same VAW-derived instances our scorer sees, with an equal GPU-hour budget and the same validation-based threshold calibration. Beating it means higher mask AP and lower head drift on the abstract-modifier split at matched or better performance on the concrete split. There is no reported result on Lacuna for SAM 3 on abstract-modifier queries; our benchmark supplies the first.
Crop-rescoring pipeline. This is the most dangerous competitor, because it shares our factorization. SAM 3 proposes head instances. Each masked crop is then scored by CLIP against the full phrase. We also run a variant that scores with LABCLIP's linear text adapter, a 512×512 adapter trained with hard negatives that raises CLEVR binding accuracy from 58% to 95% (Koishigarina et al.). Both variants use the same proposals as our method, so any difference comes from the selection step alone.
Reasoning agent. RSAgent, a multimodal LLM that calls SAM2 and zoom tools over several turns, reports 66.5% gIoU on ReasonSeg and 81.5% cIoU on RefCOCOg. It runs with its released policy. We report accuracy and wall-clock time per query.
Ablations. Four ablations isolate the key ingredients. Removing $A(t_h)$ reduces the modifier to a fixed direction. Removing $c_i$ tests whether scene context matters. Letting the modifier enter mask formation, by prompting the proposer with the full phrase, tests whether routing the modifier away from pixels is what helps. Finally, we run the full holistic segmenter with an oracle head-category filter, which separates the harm from head drift from the harm from poor selection.
Why not just…?
Why not just prompt SAM 3 with the whole phrase and fine-tune it? Scale alone has not fixed this: SAM 3 was trained on 38M synthetic phrases and still reports failure on multi-attribute queries. Whether fine-tuning closes the gap is an empirical question. The matched-budget fine-tuned SAM 3 baseline answers it. Our prediction is that it improves concrete modifiers but keeps its head drift on abstract ones, because text signal still leaks into mask formation. If fine-tuning erases the head-drift gap, the architectural claim fails and we would report that.
Why not just segment the head and rescore crops with CLIP? This is a real threat, and it is the reason the proposal is a head-conditioned, context-aware scorer rather than "segment the head, then filter". There are two reasons to expect crop rescoring to fall short. Cross-modal CLIP matching behaves like a bag of words (Koishigarina et al.). Cropping also removes the scene evidence that abstract states such as "abandoned" depend on (Hu et al.). Even so, if the LABCLIP-rescored variant matches our scorer, the honest conclusion is that the factorization alone carries the result. The context and head-conditioning ablations will show which ingredient matters.
Why not just hand everything to a multimodal LLM agent? RSAgent can reason about intent. But it is bounded by what its class-agnostic proposal tool can outline, and it needs several model calls per query. Seg-ReSearch, which interleaves reasoning with web search, averages 2.5 search turns per query. We compare accuracy per second. An agent that wins on accuracy at orders-of-magnitude higher cost would sit alongside the scorer on the cost–accuracy curve rather than replace it. An agent that wins at comparable cost would undercut the proposal.
Why not just train with hard negative captions? CLIP Won't Learn Object-Attribute Binding finds that NegCLIP-style hard negatives give only marginal binding gains on natural data. The fine-tuned SAM 3 baseline already includes hard-negative instances drawn from VAW's negative labels, so this alternative is tested directly rather than argued away.
Proposed Experiments
The main benchmark is a new abstract-modifier segmentation set built from VAW and Visual Genome instances. Masks come from box-prompted SAM and are human-verified on the test split. Modifiers are binned by collected concreteness ratings. Two held-out splits test generalization. One holds out modifiers entirely, such as "neglected" and "festive". The other holds out head–modifier pairs, to test whether $A(t_h)$ generalizes to new combinations the way EVA's polysemy results suggest it should. A small figurative-compound split, "dark horse" style, serves as a declared failure case.
The metrics are mask AP and gIoU per concreteness bin, the head-drift rate, precision at matched recall against the head-only floor, and latency.
We also run three checks for side effects and concrete-query performance. First, we evaluate RefCOCOg against PLAC's reported 41.2% Prec@1 in detection and RSAgent's 81.5% cIoU. Second, we run LVIS and SeginW to confirm that the frozen proposer's concrete-noun behaviour is untouched. Third, we evaluate on AttrSeg's Fantastic Beasts set, an unnameable-creature benchmark where name-only models score below 10 mIoU and AttrSeg reaches 58.7. It probes the opposite case, a head noun the model has never seen, where our method should be weak.
Three outcomes decide the claim:
- On the abstract bins, the factorized scorer beats both full-phrase SAM 3 and LABCLIP crop rescoring.
- Its advantage over holistic encoding grows monotonically as concreteness falls.
- Most of that advantage disappears when full-phrase SAM 3 is given an oracle head filter.
That last pattern would show head drift is the mechanism. If the gap is flat across concreteness, or if crop rescoring ties the full scorer, we would drop the asymmetry claim in favour of the simpler pipeline.
References
- "Covid vaccine is against Covid but Oxford vaccine is made at Oxford!" Semantic Interpretation of Proper Noun Compounds
- Can Large Language Models Interpret Noun-Noun Compounds? A Linguistically-Motivated Study on Lexicalized and Novel Compounds
- How Naked is the Naked Truth? A Multilingual Lexicon of Nominal Compound Compositionality
- Multi-Level Visual-Semantic Alignments with Relation-Wise Dual Attention Network for Image and Text Matching
- Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning
- SAM 3: Segment Anything with Concepts
- Language-driven Semantic Segmentation
- How do Transformer Embeddings Represent Compositions? A Functional Analysis
- CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
- Decoupling Primitive with Experts: Dynamic Feature Alignment for Compositional Zero-Shot Learning
- Exploring Vector Space Models to Predict the Compositionality of German Noun-Noun Compounds
- An Empirical Study on Compositionality in Compound Nouns
- Meta Compositional Referring Expression Segmentation
- Learning to Predict Visual Attributes in the Wild
- RSAgent: Learning to Reason and Act via Multi-Turn Tool Invocations for Text-Guided Segmentation
- Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
- CLIP Won't Learn Object-Attribute Binding from Natural Data and Here is Why
- AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation