Synthetic Crowds for Constant-Time Pose Estimators

Most multi-person pose estimators first detect each person and then run a pose network on every crop, so their running time grows with the number of people in view. One-stage models predict every person's skeleton in a single forward pass and run in roughly constant time. The scenes where that property matters most are subway platforms, stadiums, and rallies. Those dense crowds are also where one-stage models are weakest and where real training labels are scarcest. Annotators cannot see occluded joints, and most real images contain few overlapping people: nearly two-thirds of COCO images have no overlap at all, according to CrowdPose. Rendered humans solve both problems on paper. A renderer can produce any crowd density, and it knows the exact location of every hidden wrist and ankle. The cost is the synthetic-to-real gap: networks trained on rendered images pick up rendering artifacts as shortcuts and then fail on real photos.

We propose to paste photorealistic rendered humans into real crowd images at controlled densities and use them to train a fast one-stage 2D pose estimator. The study is designed to separate two things that existing work mixes together. The first is the value of more overlap. The second is the value of labels for joints no human annotator could see. A small factorial design measures each against the cost of the synthetic appearance gap. The headline measurement is whether accuracy still falls as crowd density rises, while inference latency stays flat.

Core Claim

We claim that much of the accuracy loss of one-stage estimators in crowds is a supervision problem rather than an architectural one. The problem is concentrated on occluded joints. If so, real images containing composited synthetic people, with every joint of the synthetic people supervised (hidden ones included), should flatten the curve of accuracy against crowd density. This should hold beyond what pasting extra real people achieves, and it should cost nothing at inference.

The claim also takes a position on a tension in the literature. Learning to Train with Synthetic Humans found that synthetic humans fail as training targets (13.4 mAP on MPII when trained purely on synthetic data) but help as occluders whose own loss is masked out (77.7 to 79.1 mAP). BEDLAM, a synthetic dataset rendered in Unreal Engine 5 with physically simulated clothing, showed the opposite: synthetic people can be the only targets for 3D body estimation (43.0 mm PA-MPJPE on 3DPW). We test which result holds for a constant-time multi-person head, and whether the answer changes for occluded versus visible joints.

Research Program

The base model is RTMO, a YOLO-style one-stage estimator. RTMO does not regress each coordinate directly. It classifies coordinates into bins placed inside each predicted person box, and it trains with a likelihood loss that learns a per-joint uncertainty. RTMO reports 74.8% AP on COCO at 141 FPS on a V100 and 73.2% AP on CrowdPose, but it was trained and evaluated only on real data. Its bin classifier and learned uncertainty are exactly the components that could overfit sharp synthetic edges, which makes RTMO a good test case for the gap.

We render SMPL-X bodies (a standard parametric human body mesh) using BEDLAM's body shapes, skin textures, clothing, and motions. Each body is rendered with an alpha mask and with exact 2D projections of all joints, including self-occluded ones. The rendered bodies are pasted onto real COCO and CrowdPose training images.

A placement sampler controls how much people overlap. It targets a uniform distribution over CrowdPose's Crowd Index, which measures how many of a person's joints fall inside other people's boxes. The sampler also places synthetic bodies in front of and behind real annotated people. Real people's joints that become hidden behind a pasted synthetic body keep their original labels and are marked occluded. This gives the model real-appearance supervision for occlusion.

We compare four ways of supervising the pasted bodies. In masked-occluder mode, the synthetic people get no loss, as in Hoffmann et al. In visible-joint mode, the synthetic people are supervised only on joints a human annotator could see. In full mode, every synthetic joint is supervised, hidden ones included. In uncertainty-gated full mode, hidden synthetic joints are supervised, but their loss is down-weighted when the model's learned uncertainty on a held-out real crowd set is miscalibrated for the same joint type.

The learned uncertainty also serves as a diagnostic of the domain gap. We compare predicted uncertainty against actual error on synthetic versus real occluded joints. Overconfidence on real data that does not appear on synthetic data is direct evidence that the bin classifier has learned rendering shortcuts.

An extension to constant-time 3D estimation using Single-Stage is Enough is deliberately deferred. Its depth branch infers depth from apparent body size, which adds a scale shift on top of the appearance gap. BEDLAM's barefoot subjects make that shift worse. We would pursue it only if the 2D results are positive.

Baselines

The naive reference point is RTMO-l trained on COCO and CrowdPose with its standard augmentations. Its reported 73.2% AP on CrowdPose (RTMO) is our anchor number. We retrain it ourselves so every arm shares the same code, schedule, and number of images seen.

The strongest competitor is real copy-paste. Real people are cut out of COCO using its instance masks and pasted with the same density sampler. This arm produces the same amount of extra overlap with no appearance gap. Its labels cover only joints that were visible in the source image, so it isolates our key ingredient.

The masked-occluder arm reproduces the Hoffmann et al. recipe on this architecture. The synthetic-only arm trains RTMO purely on BEDLAM-style composites, which tests whether the 13.4 mAP collapse recurs in a modern one-stage head.

All arms share the backbone, iteration count, image resolution, and hyperparameter search budget, with an equal number of trials per arm. Beating the copy-paste baseline means a statistically reliable gain, measured across three seeds, on CrowdPose's hard (high Crowd Index) subset and on OCHuman, with no loss on COCO. For reference, the specialized crowd method in CrowdPose reports 66.0 mAP on the CrowdPose test set. We found no reported result on Lacuna for RTMO on OCHuman, or for RTMO broken down by Crowd Index, so we will measure both.

Why not just…?

Why not just paste real people instead of rendered ones? This is the real threat, and it is why real copy-paste is the main baseline. If it matches full synthetic supervision, then the benefit is plain overlap and our claim fails. The visible-joint synthetic arm settles the question. It uses the same density with synthetic appearance but without the extra hidden-joint labels. Comparing it with real copy-paste measures the cost of the appearance gap. Comparing it with full mode measures the value of labels for hidden joints.

Why not just fine-tune on more real crowd data, or pseudo-label crowds with a heavy teacher? Real crowd annotations omit or guess occluded joints, and a teacher inherits the same blind spots. CoMotion pseudo-labeled large real video collections and still needed synthetic BEDLAM and WHAC-A-MOLE data for clean supervision. As an extra arm, we include pseudo-labels on CrowdPose produced by a top-down teacher. Its results on occluded joints specifically will show whether teacher labels close the gap.

Why not just use a batched top-down pipeline? Top-down systems such as RTMPose are accurate, but their cost grows with the number of people. Single-Stage is Enough reports 56 ms versus 250 ms for 20 people. We report latency as a function of person count for every arm to confirm that our training changes leave inference flat.

Decisive Checks

The primary result is AP plotted against Crowd Index on CrowdPose, together with OCHuman AP and COCO AP. Each is broken down into visible-joint and occluded-joint keypoint similarity.

The claim holds if three things are true. Full synthetic supervision must reduce the slope of the AP-versus-Crowd-Index curve relative to real copy-paste. The improvement must come mainly from occluded joints. COCO AP must not fall.

The claim is falsified under either of two outcomes. One is that visible-joint and full mode perform equally, meaning hidden-joint labels add nothing. The other is that the uncertainty diagnostic shows overconfidence on real occluded joints that grows with the share of synthetic data, meaning the bin classifier is learning rendering shortcuts.

Throughout, we measure frames per second at person counts of 1, 10, 30, and 60 to confirm that inference time stays constant.