In traditional computer vision, the task of "referring expression comprehension" (REC) or segmentation (RIS) has long operated under a simplifying assumption: one natural language description corresponds to exactly one object in an image. While this constraint made early benchmarks manageable, it created a fundamental mismatch between how AI models process scenes and how humans actually use language.
Recent research has identified this "one-to-one bias" as a primary bottleneck, preventing models from handling the complexity of real-world environments where a single phrase might refer to a group of objects, a specific set of individuals, or even nothing at all.
The Single-Instance Bottleneck
Most existing visual grounding architectures are structurally predisposed to find a single target. As noted in DetermiNet, many models are designed to output exactly one bounding box per query. This architectural choice makes it impossible for them to resolve plural quantifiers (like "those chairs") or collective references.
This limitation extends to the internal representations of these models. Jang et al. argue that representing an entire segmentation task with a single output token creates a bottleneck. When a model is forced to compress its understanding into one token, it often defaults to under-segmentation—isolating only a single "best" match and ignoring secondary objects mentioned in the instruction, a phenomenon also observed by Hu et al..
Generalized Referring Expression Segmentation
To address these failures, researchers have proposed Generalized Referring Expression Segmentation (GRES). As defined by Liu et al., GRES relaxes the traditional constraints to support an arbitrary number of targets. A model must now determine if a description refers to:
- One object: The classic case.
- Many objects: A group or set defined by shared attributes.
- Zero objects: A "rejection" case where the description does not match anything in the visual field.
This "one-to-zero" capability is crucial for robustness. Without it, models suffer from "multi-target referential ambiguity," where they are forced to hallucinate a target even when one does not exist. The OmniLabel benchmark, introduced by Schulter et al., formalizes this by requiring models to handle free-form text that may refer to any number of objects, pushing them toward true semantic understanding rather than simple noun-matching.
Building Intuition: How Generalized Grounding Works
If the problem is a fixed output structure, the solution lies in decoupling and multi-query mechanisms.
Instead of a single "target" query, newer frameworks like the Multi-Query Decoupled Interaction Network (MDIN) proposed by Wu et al. use multiple text-driven queries to probe the visual scene. By decoupling the interaction between language and vision, the model can identify multiple distinct instances in parallel.
Similarly, Dai et al. introduced InstanceVG, which uses "instance queries" to ensure that bounding boxes and segmentation masks for the same object remain consistent across complex scenes. This shift allows the model to treat each potential match as an independent hypothesis rather than trying to fit all visual evidence into a single global mask.
Beyond Static Images: Tracking and 3D Scenes
The need to break the one-to-one assumption is even more acute in dynamic or spatial contexts. In video, the Referring Multi-Object Tracking (RMOT) task requires systems to track all objects matching a description over time. Wu et al. demonstrate that traditional "detect-then-track" pipelines fail here because they cannot effectively fuse linguistic cues across multiple simultaneous targets.
In 3D environments, Abdelreheem et al. show that natural language often uses "anchor objects" to describe sets (e.g., "the two chairs near the table"). By training models to recognize these one-to-many correspondences, systems achieve a more nuanced understanding of spatial relationships and scene layout.
The Role of Linguistic Logic
Finally, moving beyond the one-target constraint requires models to understand the logic of determiners—words like "some," "all," or "those." DetermiNet reveals that while current models are excellent at identifying object categories (e.g., "finding a dog"), they struggle with the linguistic logic required to select the correct set of objects.
By transitioning toward multi-region grounding, as advocated by Chen et al. for assistive technologies, AI can begin to answer comparative or distributive questions that are essential for real-world utility, such as identifying all individuals in a crowd who match a specific description (Referring to Any Person).
Key Foundational Work
Image description with a goal: Building efficient discriminating expressions for images (2012) — contributed 1 concept elements to this direction.
Graph-Based Pancreatic Islet Segmentation for Early Type 2 Diabetes Mellitus on Histopathological Tissue (2009) — contributed 1 concept elements to this direction.