Flowing Crowd Density into Detections

Detectors that work well on ordinary photos break down in crowds. When people overlap heavily, two boxes that should both survive look like duplicates and get deleted, or one box absorbs two people. A recent line of work treats detection as generation. A diffusion model, which is a generator that learns to turn random noise into data by removing noise step by step, "denoises" a set of random box positions into the final detections. This helps, but it has an odd defect. DiffuDETR runs its sampler deterministically, as an ordinary differential equation (ODE), which gives a fixed noise-free path from start to answer. Yet its accuracy plateaus or drops when the number of solver steps rises past about three. In image generation, more ODE steps reliably help, so this behaviour points to a mismatch between how the model is trained and how its paths are integrated at test time.

We propose to start each detection's path from a sample of a predicted crowd-density map rather than from Gaussian noise. The decoder is then trained as a flow-matching velocity field, meaning a network that predicts at each moment which way and how fast each box should move. It learns to carry those density samples along short, nearly straight, non-crossing paths to individual people. Paired training targets come from optimal transport, a matching that pairs each start with a target so that total travel distance is as small as possible. Along the way, the moving boxes attend to every other moving box, so they can see that a person is already "taken" and steer away. The bet is that this turns the step-count problem into a monotone accuracy-versus-compute curve. It should also replace greedy duplicate suppression with a continuous transport of density mass into separated instances.

Core Claim

Our claim is that DiffuDETR's step collapse is caused by two things. First, its starting points are image-independent Gaussian noise. Second, its training pairs noise with targets arbitrarily, which forces curved, crossing paths that a few large steps happen to approximate better than many small ones. If both causes are removed, crowded-scene detection accuracy should rise or stay flat as solver steps increase from 1 to 20. Removing them means starting from the image's own density field and training with an optimal-transport pairing that makes paths short and straight. The flow must also let every moving query attend globally to every other one.

Concretely, let $x_0$ be a set of $K$ start boxes. Each has a center drawn from the predicted density map $\hat{D}$ and a width and height read from a predicted scale map at that center. Let $x_1$ be the ground-truth boxes, reordered by an optimal-transport assignment $\pi$ between start centers and true centers. Unmatched starts are paired with themselves and labeled background. Training uses the standard straight-line flow-matching objective:

$$ \mathcal{L} = \mathbb{E}_{t \sim U[0,1]} \left\| v_\theta\big((1-t)\,x_0 + t\,x_{1}^{\pi},\; t,\; F\big) - (x_{1}^{\pi} - x_0) \right\|^2 , $$

Here $v_\theta$ is the decoder viewed as a velocity network, $F$ is the image feature map, and $x_1^{\pi}$ is the ground truth after reordering. At test time we draw $K = \lceil \sum \hat{D} \rceil + m$ starts, where $m$ is a small safety margin, and integrate $dx/dt = v_\theta(x,t,F)$ with Euler or Heun steps. A classification head on the final boxes decides foreground versus background. No non-maximum suppression (NMS), the greedy post-processing step that deletes lower-scoring boxes overlapping a higher-scoring one, is used.

Two things make this different from DiffuDETR. The randomness lives entirely in where the starts fall, and those positions come from the image's density field. The training pairs are also geometrically consistent, which is what should make more steps help instead of hurt.

Research Program

The work has three parts, each tied to one of the four input directions.

Density-informed starting points. Solving Diffusion ODEs with Optimal Boundary Conditions showed that in a deterministic sampler, the starting state is the only source of randomness. Choosing it well gives better super-resolution in far fewer steps: 100 DDIM steps reach an LPIPS of 0.1032 on DIV2k, against 0.1075 for 1000 stochastic DDPM steps. Their optimal start is a single fixed tensor tied to one image size. That paper supplies our method, and we extend it. Our start is input-dependent and variable-sized: it is a point set sampled from a density map, with as many points as the map's integral. The density head follows the counting literature (MCNN regresses continuous head densities). It is supervised with the same multi-level density targets used by Density-based Object Detection in Crowded Scenes.

Transport as the detection mechanism. Integer programming on object density maps recovers discrete instances from density by solving an integer program under count constraints. It works well on merged cells (F1 93.67%), but the program is NP-hard and sits outside the network. Density-guided anchors use unbalanced optimal transport only as a training-time assignment and fall back to a density-adaptive NMS at test time. Both papers motivate us, because they show that density plus a global count constraint separates merged instances. We keep the optimal-transport idea and move it into the sampler: transport is the inference procedure, learned end to end. It is solved once per training image on center coordinates, using standard entropic optimal transport.

Local versus global interaction in the velocity network. This is where the receptive-field direction enters as a design constraint rather than as backbone engineering. Progressive End-to-End Object Detection in Crowded Scenes refines uncertain queries with self-attention restricted to spatial neighbors, which brings back a local receptive field at the query level. We hypothesize that exclusion in crowds is not purely local. Whether a start should go to person A or the adjacent person B depends on where all the other starts are heading, and density mass that leaves a dense cluster must be accounted for globally. The velocity network therefore uses full self-attention among the $K$ moving queries, which is cheap at a few hundred tokens, plus deformable cross-attention to image features.

Dense attention over image pixels is the costly part. Its quadratic memory is a documented bottleneck in Differentiable Spatial Planning using Transformers. The density head therefore gets image-wide context through recurrent criss-cross attention from CCNet, which routes information along rows and columns and reports 11x less memory than a non-local block. This matters because a local convolutional density head can mistake background clutter for crowd. Any such false density mass becomes wasted starts.

We will also measure path straightness, defined as the ratio of a path's chord length to its integrated arc length. This directly tests the mechanism claimed for step-monotonicity, not just the final accuracy.

Baselines

The naive reference point is a standard query-based detector, Deformable DETR or DINO, with its usual learned queries and Hungarian matching, run on CrowdHuman with no crowd-specific changes. It anchors how much any crowd method buys. The FPN baseline in CrowdHuman has roughly a 50% miss rate, which shows how badly unmodified detectors fare.

The most direct competitor is DiffuDETR, which is also an ODE-sampled query detector. It reports 51.9 mAP on COCO with ResNet-50, 1.0 above DINO. No reported result was found on Lacuna for DiffuDETR on CrowdHuman, so we will reimplement it from its paper with our backbone and training schedule. We will report its accuracy at 1, 3, 5, 10 and 20 DDIM steps. Beating it means matching or exceeding its best-step accuracy while showing a non-decreasing curve as steps increase.

The strongest crowd-specific competitors are the progressive query method and density-guided detection. On CrowdHuman, the progressive method on Sparse R-CNN reports 92.0 AP (Progressive End-to-End Object Detection). Density-guided anchors with density-adaptive NMS take FCOS from 86.8 AP and 54.0 miss rate to 89.5 AP and 44.1 miss rate, and reach 92.2 AP when combined with MIP (Density-based Object Detection in Crowded Scenes). We will cite these published numbers. We will also retrain the progressive method on our backbone and epoch budget, since published numbers mix backbones and schedules. Beating 92.2 AP outright is a stretch goal. The success criterion is to exceed the retrained progressive method at equal backbone, epochs and FLOPs on both AP and log-average miss rate, with the clearest gains on the subset of person pairs whose boxes overlap with IoU above 0.5.

Three ablations isolate the key ingredient. The first keeps density starts but drops the ODE: a density-seeded decoder of equal depth trained with Hungarian loss. The second keeps flow matching and optimal-transport pairing but starts from Gaussian noise. The third keeps everything but restricts query-query attention in the velocity network to local windows, mirroring the progressive method's local attention. All variants share the backbone, the density head, the training epochs, and a fixed hyperparameter search budget of the same number of runs.

Why not just…?

Why not just seed a standard DETR decoder with density peaks, with no ODE at all? This is the most serious threat. DINO-style detectors already pick starting anchors from encoder scores, and density-seeded queries plus Hungarian training could capture most of the gain. Our reason to doubt it is the documented saturation of query decoders in crowds: deeper decoders keep producing duplicate high-confidence boxes (Progressive End-to-End Object Detection). A fixed stack of layers trained with Hungarian matching also gives no notion of more compute meaning better answers. The first ablation settles this directly. If density-seeded Hungarian decoding matches our flow at its best step count, the honest conclusion is that the starting points, not the ODE, did the work. We would then report the proposal as a query-initialization result.

Why not just fix DiffuDETR by tuning its noise schedule or always using three steps? Fixing the step count works in practice, but it treats the symptom and gives up the main promise of deterministic samplers. Progressive distillation and SinSR show that that promise is a fixed, well-behaved path one can trade steps along or distill. The second ablation, Gaussian starts with optimal-transport pairing, tests whether pairing alone repairs the step curve. If it does, density starts would be shown to add only accuracy, not stability, and we would say so.

Why not just use density-adaptive NMS or Soft-NMS on a strong detector? Density-guided NMS is cheap and works: it is half of the 54.0 to 44.1 miss-rate improvement on FCOS. But it still deletes boxes by thresholding overlap. CrowdHuman averages 2.4 true pairs per image with IoU above 0.5, which no overlap threshold can tell apart from duplicates (CrowdHuman). We will report recall on exactly these high-overlap pairs. That subset is where the proposal must win; if it does not, the added complexity is not justified.

Why not just use local attention among queries, which is cheaper and was enough for the progressive method? At a few hundred queries, global attention costs little, so the question is accuracy, not compute. The third ablation tests it. We expect the largest gap in large dense clusters, where a start must know about mass far outside its window.

Proposed Experiments

The main benchmark is CrowdHuman, using full-body boxes. We report AP, log-average miss rate, and recall on true pairs with IoU above 0.5, using ResNet-50 for all methods. CityPersons tests transfer to moderate crowding, where density guidance reports 96.7 AP and 10.0 miss rate. COCO 2017 checks that the method does not hurt sparse scenes; there the bar is DiffuDETR's 51.9 mAP. Because density starts are less informative when scenes contain many object classes, the method will use one class-agnostic density map, and a COCO drop of more than about 0.5 mAP would mark a real limitation. To confirm that the density head counts sensibly, we also report counting error on ShanghaiTech Part A. MCNN's reported mean absolute error of 110.2 is a floor rather than a target. UCSD pedestrians link the method to the integer-programming line, which reports about 91% F1.

Three checks decide the proposal. First, the AP-versus-steps curve from 1 to 20 Euler steps must be non-decreasing for our method on CrowdHuman, while our DiffuDETR reimplementation reproduces its reported saturation; together with the straightness measurements, this tests the mechanism. Second, the density-seeded Hungarian ablation must trail the full method on high-overlap recall; otherwise the ODE is unnecessary. Third, the image-wide criss-cross density head must cut false density mass on background clutter compared with a convolutional head of matched size, measured as the integral of predicted density outside annotated regions. Global attention among queries must also beat windowed attention on the largest clusters. If all three hold, the result is a crowd detector whose accuracy can be bought with solver steps and whose duplicate handling is learned transport rather than a threshold.