TL;DR
Traditional vision-language grounding treats text as a fixed query to localize an image region. This paper introduces bidirectional concept correspondence, a unified formulation that jointly discovers which text spans are referential and how they map to instance-level image segments. Their model, ConCor-1, leverages learnable "bridge tokens" on top of a vision-language model (VLM) to achieve massive performance gains, including a 48% improvement in JointF1 on dense caption datasets.
What Problem Does This Paper Address?
Visual grounding is crucial for making multimodal systems interpretable and actionable. However, existing setups (like phrase grounding or open-vocabulary detection) simplify the problem into a one-way street: given a pre-specified text phrase, find the corresponding pixels.
This unidirectional approach overlooks a fundamental challenge in human communication: listeners aren't handed pre-segmented text labels. In the real world, a model must look at an image-text pair and figure out which parts of the text are visually referential, resolve coreferences (e.g., realizing that "dog," "it," and "the furry animal" all point to the same entity), and map them to distinct visual objects.
The authors address this gap by unifying text segmentation, image segmentation, and cross-modal alignment into a single, bidirectional correspondence problem.
Methods
The authors introduce ConCor-1, a model built on top of the Qwen3.5-0.8B VLM backbone. Instead of using the VLM for autoregressive text generation, ConCor-1 treats it as a contextual encoder and introduces a non-autoregressive prediction architecture.
The Secret Sauce: Bridge Tokens
To represent candidate image-text correspondences, the model appends a set of 385 learnable bridge tokens to the multimodal input sequence. Using bidirectional attention, these tokens interact simultaneously with both visual and textual tokens, aggregating the context needed to hypothesize potential alignments.
Multi-Scale Spatial Assignment
To force different bridge tokens to specialize in different types of objects, the authors organize them into multi-scale spatial grids (ranging from $1 \times 1$ to $10 \times 10$). During training, Hungarian matching pairs ground-truth image masks with the bridge tokens based on spatial grid overlap.
Lightweight Prediction Heads
Three task-specific heads process the final hidden states of the bridge tokens:
- Presence Head: A simple MLP that predicts a score indicating whether the bridge token represents a valid, real correspondence.
- Text Segmentation Head: Computes a bilinear compatibility score between the bridge token and text tokens to output a binary mask over the text sequence (allowing for non-contiguous, coreferential spans).
- Image Segmentation Head: Reconstructs dense visual features by fusing VLM tokens with raw vision encoder features, passing them through a convolutional upsampling decoder to output fine-grained instance masks.
Datasets
Because standard datasets do not natively support coreference-aware bidirectional grounding, the authors engineered an LLM-based rewriting pipeline to clean, resolve, and standardize existing data into a unified correspondence format.
- Caption Grounding Data: GoldG (derived from Flickr30k/GQA), COCONut-PanCap (dense panoptic captions), and GroundedRef (a newly constructed validation/training split featuring complex, synthetic referring expressions generated from COCO panoptic segments).
- Instance Segmentation Data: Standard benchmarks including COCO, COCONut, EntitySeg, ADE20K, PixMo-Point, and SA-1B were integrated to provide robust, large-scale mask supervision.
Key Insights and Ideas
- Bridge Tokens as Modality Binders: Instead of relying on complex cross-attention matrices or fragile text generation, explicit query tokens can cleanly encapsulate a multi-modal concept, capturing non-contiguous text spans and visual masks simultaneously.
- Progressive Grounding Emergence: By probing the VLM layers, the authors discovered that grounding capability isn't present in early layers. Instead, bridge tokens progressively accumulate cross-modal correspondence information as they pass deeper through the language model.
- Single-Pass Large Vocabularies: Traditional open-vocabulary detectors struggle with massive category lists (like LVIS) and require chunking the text into multiple forward passes. ConCor-1 can ingest an entire category list as a single text sequence, resolving all alignments at once.
Key Results
ConCor-1 was evaluated using JointF1, a strict metric requiring both the text span IoU and the image mask IoU to exceed 0.5 for a prediction to be counted as correct.
- Dense Captions: On the complex COCONut-PanCap dataset, ConCor-1 achieved a 88.8 JointF1, outperforming the strongest baseline (GPT-5.4 + SAM) by 48% relative.
- Zero-Shot Category Grounding: On LVIS-minival, ConCor-1 achieved a 29.9 JointF1, representing a 29% relative improvement over the next best baseline (GLaMM), while processing the entire vocabulary in a single forward pass.
- Qualitative Disambiguation: Qualitative evaluations demonstrated that ConCor-1 successfully resolves ambiguous scenarios where multiple similar objects or complex contextual relations are present—areas where traditional detector-based baselines (like Grounding DINO + SAM) frequently fail.
Limitations and Future Work
- Mask Quality vs. Specialized Models: Because the image segmentation head is intentionally lightweight to prioritize joint correspondence efficiency, its raw mask boundaries still lag slightly behind heavy, specialized segmentation-only models.
- Supervision Dependency: The framework relies heavily on paired, coreference-mapped text and image data. Converting standard single-modality datasets into this format can introduce noise.
- Future Horizons: The authors propose extending this bidirectional correspondence framework to video data (where alignments must persist across time dimensions) and utilizing it as an automated metric to evaluate the semantic faithfulness of text-to-image generative models.
Summary updated