Constant Time Inference for Multi Person Vision

This direction is split into other directions: linear inference scaling. Its last version stays readable and citable here.

Version 1 of 1180 papers192 conceptsfrozen 2026-10-04

In computer vision, the difficulty of a task usually scales with the complexity of the scene. For years, the standard approach to analyzing multiple people—whether estimating their poses, tracking their movements, or identifying their gaze—followed a "top-down" logic: first, detect every person in the frame; second, crop the image around each person; and third, run a specialized model on each crop.

While intuitive, this approach harbors a fundamental flaw: its computational cost scales linearly ($O(N)$) with the number of people. In a crowded subway station or a busy street, a system that works perfectly for one person becomes prohibitively slow as the crowd grows.

A growing body of research is shifting toward "one-stage" or "bottom-up" architectures. These methods aim for constant-time inference, where the cost of processing a frame remains largely independent of how many people are in it.

The Top-Down Bottleneck

To understand why the field is moving away from sequential pipelines, we must look at the "crop-and-process" tax. As noted by Tyler Zhu in Towards Accurate Multi-person Pose Estimation in the Wild, top-down frameworks are highly accurate but face a significant bottleneck in crowded environments. Every additional person requires a new inference pass.

This linear scaling isn't just a minor lag; it is a barrier to real-time interaction. Mingfang Zhang points out in GazeOnce: Real-Time Multi-Person Gaze Estimation that traditional gaze estimation cannot maintain the frame rates required for augmented reality or public displays when more than a few users are present. The goal of this research direction is to decouple the "what" (the analysis) from the "how many" (the instance count).

Bottom-Up Grouping: Connecting the Dots

The first major alternative to the top-down paradigm was the "bottom-up" approach. Instead of finding people and then finding their parts, these models find all the parts in an image simultaneously and then group them.

A seminal example is the work on Part Affinity Fields (PAFs) by Zhe Cao, Tomas Simon, and colleagues in Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. Instead of processing people, the network predicts vector fields that encode the association between body parts (e.g., "this elbow belongs to this shoulder"). Because the network only scans the image once to find all parts and all associations, the inference time remains nearly constant regardless of the crowd density.

This philosophy has been extended to other domains, such as facial analysis. Kele Xu and Bin Huang developed FoxNet, which uses a bottom-up approach for multi-face alignment. By treating landmarks as points to be clustered rather than properties of a detected box, they achieved high-speed performance that doesn't degrade in large groups.

One-Stage Regression: From Pixels to Meshes

While bottom-up methods solve the scaling problem for 2D points, 3D reconstruction—recovering the full volume and shape of multiple people—traditionally remained stuck in the top-down loop.

Yu Sun, Michael J. Black, and Tao Mei challenged this with ROMP. Instead of cropping people, ROMP regresses 3D body meshes directly from a pixel-level representation. By representing people as "body centers" on a feature map, the model can "read out" the 3D parameters for every person in a single forward pass. This allows for real-time 3D mesh recovery (30+ FPS) even in scenes with heavy occlusion where bounding boxes often fail.

Similarly, in the realm of gaze following, Danyang Tu and Wei Shen introduced GTR. By replacing multi-stage cascaded pipelines with a unified architecture, they achieved a 9x speedup, proving that the "where" (head location) and the "what" (gaze target) can be solved simultaneously without linear overhead.

Unified Transformers and Global Context

The most recent evolution in this direction leverages Transformers to treat instances as tokens. In traditional CNNs, the receptive field is limited, making it hard to associate a person on one side of the image with an object on the other without multiple stages.

Transformers solve this by allowing every part of the image to attend to every other part. In Sharingan, Samy Tafasca and Jean-Marc Odobez treat people as special "gaze tokens." These tokens interact with the rest of the image through self-attention, allowing the model to predict the gaze targets of everyone in the scene at once.

This "whole-image" strategy is also central to the work of Serena Yeung-Levy and Zhenzhen Weng. In Multi-Human Mesh Recovery with Transformers, they demonstrate that processing the entire image at once isn't just faster—it's more accurate. By avoiding individual crops, the model retains the global context necessary to understand the relative positioning and interactions between people, which is often lost in top-down methods.

Why It Works: Shared Computation

The core intuition behind all these methods is computational reuse. In a top-down system, if two people are standing next to each other, the system processes the background and overlapping pixels twice. In a unified system, the feature extraction happens once.

Whether it is JDE (Joint Detection and Embedding) by Zhongdao Wang for tracking, or TRFH by Shicun Chen for face detection, the goal is the same: move the "instance-specific" work into a single, shared forward pass. By treating people as features of the image rather than independent images themselves, these researchers are enabling vision systems that can finally keep pace with the complexity of the real world.

Papers in this version

Showing 60 of 180 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_constant_time_inference_for_multi_person_6afb171c,
  title = {Constant Time Inference for Multi Person Vision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/constant-time-inference-for-multi-person-vision/rdr_6afb171c35334c08858ccf59420ad752},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_constant_time_inference_for_multi_person_6afb171c_v1,
  title = {Constant Time Inference for Multi Person Vision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/constant-time-inference-for-multi-person-vision/rdr_6afb171c35334c08858ccf59420ad752/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}