In computer vision, the most intuitive way to estimate the poses of multiple people in an image is the top-down approach. This strategy breaks the problem into two sequential steps: first, run an object detector to locate every person and draw a bounding box around them; second, crop each bounding box and pass it through a dedicated single-person pose estimation network.
While highly accurate—as demonstrated by engineered pipelines like the one proposed by Papandreou et al. (Towards Accurate Multi-person Pose Estimation in the Wild) or the clean baselines established by Xiao et al. (Simple Baselines for Human Pose Estimation and Tracking)—this paradigm suffers from a fundamental architectural flaw: its computational cost scales linearly ($O(N)$) with the number of people ($N$) in the scene.
As noted by Sekii (Pose Proposal Networks), this dependency means that processing time fluctuates based on crowd density. In a crowded subway station or a busy public square, a top-down system must instantiate dozens of sub-networks sequentially. Ning et al. (A Top-down Approach to Articulated Human Pose Estimation and Tracking) and the authors of DynPose emphasize that this linear scaling creates an unpredictable latency bottleneck, making top-down frameworks highly impractical for real-time applications in dense environments.
Decoupling Complexity via Bottom-Up Inference
To achieve predictable, real-time performance, a system's execution time must be invariant to the number of people in the frame ($O(1)$ complexity). This insight drives the development of bottom-up approaches. Instead of treating people as isolated crops, bottom-up methods process the entire image in a single forward pass to detect all body joints simultaneously.
Mehta et al. (XNect: Real-time Multi-person 3D Human Pose Estimation with a Single RGB Camera) demonstrate that by identifying all body joints globally in one pass, the neural network backbone's computational overhead becomes completely decoupled from the number of individuals in the scene. Similarly, Hidalgo et al. (Single-Network Whole-Body Pose Estimation) designed a unified network capable of extracting body, face, hand, and foot keypoints at a constant frame rate, regardless of crowd size. By avoiding the need to spawn separate networks for each person, these architectures ensure stable, predictable latency.
The Grouping Bottleneck and the Rise of Single-Stage Models
While early bottom-up methods successfully decoupled the neural network's execution time from crowd density, they introduced a secondary computational trap: the grouping bottleneck. After detecting a chaotic sea of isolated joints (e.g., fifty left shoulders and fifty right elbows), the system had to figure out which joints belonged to which person.
As highlighted by Tian et al. (PoseDet: Fast Multi-Person Pose Estimation Using Pose Embedding) and Martiez-Gonzalez et al. (Real-time Convolutional Networks for Depth-based Human Pose Estimation), these grouping stages typically relied on greedy parsing algorithms or combinatorial optimization. In dense scenes, the time spent assembling these skeletons could skyrocket, defeating the purpose of the fast bottom-up backbone.
To achieve true constant-time inference, modern researchers have pioneered single-stage frameworks that eliminate both individual cropping and complex joint grouping. They achieve this through several clever mechanisms:
- Hierarchical Displacements and Embeddings: Nie et al. (Single-Stage Multi-Person Pose Machines) introduced a structured root-displacement model that directly links person centers to their respective joints in a single pass. Sekii (Pose Proposal Networks) bypassed heatmaps entirely by treating body parts and limbs as bounding box detection targets, while Tian et al. (PoseDet) utilized keypoint-aware pose embeddings to formulate pose estimation as a direct object detection problem.
- Dynamic Convolutions: Instead of cropping regions of interest, the authors of FCPose utilized dynamic, instance-aware convolution weights to generate poses directly. Lu et al. (RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation) achieved a similar feat by replacing traditional coordinate regression with a Dynamic Coordinate Classifier, maintaining high frame rates irrespective of person density.
- Anchor Regression and Set Prediction: Benzine (PandaNet: Anchor-Based Single-Shot Multi-Person 3D Pose Estimation) demonstrated that regressing full poses directly from predefined spatial anchors allows for stable, single-shot multi-person estimation. Taking this further, Wang et al. (Direct Multi-view Multi-person 3D Pose Estimation) formulated multi-person 3D pose estimation as a set-prediction problem via transformers, allowing the model to output all skeletons simultaneously in constant time.
Scaling to 3D, Tracking, and Holistic Capture
The push for constant-time inference has expanded beyond 2D keypoints into 3D human mesh reconstruction and temporal motion capture. Traditional 3D multi-person pipelines are notoriously heavy, often requiring per-instance 3D convolutions or iterative refinement loops that scale poorly in crowds, as noted by Jin et al. (Single-Stage is Enough: Multi-Person Absolute 3D Pose Estimation).
To counter this, Wang (PSVT: End-to-End Multi-Person 3D Pose and Shape Estimation with Progressive Video Transformers) designed an end-to-end progressive decoding mechanism that extracts 3D shapes without sequential cropping. For temporal tracking, Raaj et al. (Efficient Online Multi-Person 2D Pose Tracking With Recurrent Spatio-Temporal Affinity Fields) introduced Spatio-Temporal Affinity Fields (STAF), which recurrently link limbs across frames to maintain real-time tracking speeds regardless of the number of tracked subjects. Finally, Liu et al. (Motions as Queries: One-Stage Multi-Person Holistic Human Motion Capture) extended this concept to full motion capture by treating individual human motions as parallel temporal queries within a transformer architecture, bypassing separate tracking and detection modules entirely.
By shifting from sequential, instance-based processing to unified, single-stage architectures, the field has successfully decoupled computational complexity from scene density, paving the way for predictable, real-time human sensing in the wild.
Key Foundational Work
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields (2017) — contributed 2 concept elements to this direction.
Towards Accurate Multi-person Pose Estimation in the Wild (2017) — contributed 1 concept elements to this direction.
Towards Accurate Multi-person Pose Estimation in the Wild (2017) — contributed 1 concept elements to this direction.