One-Stage Visual Grounding and Monolithic Multimodal Architectures

This direction is split into other directions: single-stage visual grounding. Its last version stays readable and citable here.

Version 1 of 161 papers74 conceptsfrozen 2026-10-04

For years, the standard way to help a computer "see" what we are talking about followed a two-step recipe: first, use a pre-trained object detector to draw thousands of "proposal" boxes around everything in the image; second, use a language model to rank those boxes and find the one that matches the description.

While effective, this "detect-then-match" paradigm—often called a two-stage architecture—suffers from a fundamental "proposal bottleneck." If the initial detector fails to draw a box around the target, the language model can never find it. Furthermore, running a heavy object detector before even looking at the text is computationally expensive and slow.

A new wave of research is moving toward one-stage and monolithic architectures. These models eliminate the middleman, fusing visual and textual features into a single representation to localize objects directly from raw pixels or unified tokens.

Breaking the Proposal Bottleneck

The core intuition behind one-stage grounding is that the model should "know what it's looking for" before it starts localizing. In traditional systems, the object detector is "blind" to the text; it proposes boxes for every dog, car, and tree it sees.

In contrast, the Single-Stage Grounding (SSG) network introduced by Lin Ma and Xinpeng Chen treats localization as a direct regression task. By fusing multimodal features early, the model can predict bounding box coordinates in a single pass. This approach, as detailed in Real-Time Referring Expression Comprehension by Single-Stage Grounding Network, allows for blistering speeds of 40 referents per second, enabling real-time interaction.

This shift isn't just about speed; it's about robustness. In 3D environments, where point clouds are sparse and objects are often occluded, a generic detector might fail to propose a target entirely. Junyu Luo and Si Liu demonstrated in 3D-SPS that a single-stage approach—using language to guide the selection of keypoints directly—prevents the information loss that plagues decoupled systems.

From Modular to Monolithic

The evolution of visual grounding has recently converged with the rise of Multimodal Large Language Models (MLLMs). Traditionally, these models are "modular": they use a separate, frozen vision encoder (like CLIP) and "bolt" it onto a language decoder.

However, this modularity introduces a new kind of latency. The system must wait for the vision encoder to finish processing the entire image before the language model can generate its first word. Gen Luo and Xue Yang address this in Mono-InternVL by proposing a monolithic architecture. By integrating visual perception directly into the core transformer blocks, they reduced first-token inference latency by up to 67%.

Taking this a step further, Haiwen Diao and Xin-Long Wang introduced EVE (Encoder-free Vision-Language Model). As described in Unveiling Encoder-Free Vision-Language Models, EVE processes image patches directly within the language decoder, achieving an order of magnitude speedup in image encoding by removing the need for a standalone vision model entirely.

Why It Works: Unified Feature Spaces

The success of these one-stage and monolithic models stems from three key mechanisms:

  1. Direct Regression and Correlation: Instead of ranking boxes, models like RCCF treat language as a "filter" that slides over the image. The text is mapped to a visual kernel, and the peak of the correlation map indicates the object's center.
  2. End-to-End Optimization: By removing fixed detectors, the entire network can be optimized for the specific task of grounding. Zhicheng Huang argued in Seeing Out of the Box that this allows the model to learn visual features that are specifically relevant to the linguistic queries, rather than generic object features.
  3. Token Fetching: Modern MLLMs like ClawMachine by Tianren Ma avoid coordinates altogether. They treat regions of an image as "token collectives" that the model can "fetch" from its vocabulary, unifying the way the model "thinks" about words and pixels.

The Future of Real-Time Reasoning

The transition from two-stage to one-stage architectures represents a move toward more biological-like perception. Humans don't draw thousands of mental boxes before deciding which one is a "red apple"; we use the concept of "red apple" to direct our gaze.

By consolidating localization, reasoning, and generation into a single forward pass, researchers are creating systems that are not only faster—as seen in the 4x speedup of Type-to-Track—but also more capable of complex, open-world interaction. Whether it is an autonomous vehicle understanding a command to "Pull over next to that lady on the left" or a medical AI mapping diagnostic phrases to X-ray regions, the future of multimodal AI lies in these unified, single-stage architectures.

Papers in this version

Showing 60 of 61 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_one_stage_visual_grounding_and_monolithi_1fbc478a,
  title = {One-Stage Visual Grounding and Monolithic Multimodal Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/one-stage-visual-grounding-and-monolithic-multimodal-architectures/rdr_1fbc478ac9a6456a8019cf2d0e2c7f09},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_one_stage_visual_grounding_and_monolithi_1fbc478a_v1,
  title = {One-Stage Visual Grounding and Monolithic Multimodal Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/one-stage-visual-grounding-and-monolithic-multimodal-architectures/rdr_1fbc478ac9a6456a8019cf2d0e2c7f09/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}