Scaling Query Based Vision to Dense Scenes

This direction is split into other directions: fixed query capacity. Its last version stays readable and citable here.

Version 1 of 135 papers38 conceptsfrozen 2026-10-04

Modern computer vision has undergone a fundamental shift from predicting properties of every pixel to predicting properties of a set of objects. This transition, popularized by the Detection Transformer (DETR) and its successors, relies on "object queries"—a fixed set of learnable vectors that act as slots to be filled by entities in a scene. While this approach simplifies the vision pipeline, it introduces a rigid architectural constraint: the fixed query bottleneck.

The Fixed Query Bottleneck

In a query-based system, the model is initialized with a predefined number of queries (e.g., 100 or 300). Each query is responsible for "asking" the image if a specific object exists at a certain location or with certain characteristics. As Alex Kirillov and Bowen Cheng demonstrate in MaskFormer, this allows for a unified approach to semantic and panoptic segmentation.

However, this design assumes that the number of objects in a scene will never exceed the number of queries. When a model encounters a "dense" scene—such as a crowded street, a complex document, or a microscopic slide—this assumption fails. If a scene contains 500 people but the model only has 100 queries, 400 people will simply never be detected. This "hard upper bound," as noted by Serena Yeung-Levy in the context of multi-human mesh recovery, creates a scalability ceiling that prevents models from generalizing to high-density environments.

Where Capacity Meets Complexity

The limitations of fixed-query architectures manifest across several domains:

  • Crowd Counting and Localization: In tasks where the goal is to count thousands of instances, a fixed query budget leads to massive undercounting. Niki Amini-Naieni and Andrew Zisserman observe in CountGD++ that without complex tiling or cropping strategies, standard Transformers cannot process high-resolution images containing thousands of small objects.
  • Document Analysis: Documents like newspapers or technical manuals often contain more layout elements than a standard detector can handle. Hiuyi Cheng highlights in M6Doc that fixed-query frameworks fail to detect all relevant instances when element density is high.
  • 3D Scene Understanding: In 3D environments, the problem is compounded by spatial clutter. Yu Liu and colleagues show in SlotLifter that representing scenes through a fixed number of "slots" limits the ability to segment small, distinct entities in cluttered real-world scans.

Strategies for Infinite Scalability

To overcome these limits, researchers are moving away from static query sets toward more dynamic and dense representations.

1. Dense and Hybrid Queries

Instead of a small set of learnable vectors, some models use the image features themselves as queries. Yihong Xu and Chuang Gan introduced TransCenter, which uses dense, image-related representations for detection. By associating queries directly with pixel locations, the model can scale its detection capacity to match the image resolution, ensuring that no object is left "un-queried."

2. Adaptive Hierarchies

Another approach is to allocate queries where they are needed most. In PET, Chengxin Liu and Hao Lu 0003 propose a Point-Query Quadtree. This mechanism recursively decomposes the image into smaller regions based on density. In sparse areas, few queries are used; in dense areas, the quadtree branches out, generating more queries to handle the increased complexity. This balances computational efficiency with the need for high-granularity detection.

3. Linear Complexity Architectures

The bottleneck is often as much about memory as it is about query count. Standard Transformer attention scales quadratically with the number of tokens, making a massive query set computationally prohibitive. New frameworks like MOGO, developed by Yunqing Liu, leverage the Mamba architecture's linear complexity. By eliminating the quadratic overhead of self-attention, these models can process more information with significantly lower memory usage, allowing for more exhaustive scene parsing.

Conclusion

The shift from fixed to dynamic query systems represents a maturation of the object-centric paradigm. By treating query capacity as a flexible resource rather than a static hyperparameter, researchers are enabling models to move from simple "one-object-per-slot" reasoning to the comprehensive understanding of the complex, crowded, and cluttered environments that define the real world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_query_based_vision_to_dense_scen_78f4dda5,
  title = {Scaling Query Based Vision to Dense Scenes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-query-based-vision-to-dense-scenes/rdr_78f4dda565554a1294eabc5d37bf288c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_query_based_vision_to_dense_scen_78f4dda5_v1,
  title = {Scaling Query Based Vision to Dense Scenes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-query-based-vision-to-dense-scenes/rdr_78f4dda565554a1294eabc5d37bf288c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}