Do Crowded People Become One Query? Over-Smoothing in Transformer Detector Decoders

Transformer detectors locate objects with a fixed set of learned slots called "object queries". Each query is refined through a stack of decoder layers. In every layer the queries exchange information with one another through self-attention, and each query then reads image features. Self-attention averages: every query's new value is a positively weighted mean of all queries. Repeated averaging is what graph researchers call over-smoothing, meaning that node features drift toward one shared value as layers stack up. Two pedestrians standing shoulder to shoulder in a crowd are exactly the case where this should hurt. Their two queries look at nearly the same pixels and attend strongly to each other, and they may converge until the detector either outputs one box for both people or two duplicate boxes for one person.

We propose to test this directly on crowded-pedestrian benchmarks and, if it holds, to fix it with a targeted change. The change adds a "difference" channel to the decoder's query self-attention that keeps each query distinct from its neighbors. That channel is switched on only for queries whose predicted boxes heavily overlap other confident predictions. In plain terms, we let queries share context freely when objects are far apart and force them to stay distinct exactly where objects occlude each other.

Core Claim

The claim has two parts, and the first stands on its own. First, in end-to-end detectors run on dense crowds, the representations of queries assigned to different but mutually occluding people become measurably more similar with decoder depth. This growing similarity predicts missed and merged detections, and it is worse for small, distant people, whose image evidence is weakest. Second, a high-pass correction gated by predicted overlap reduces that collapse and the associated misses. It does so more effectively than an ungated, uniform high-pass correction, and more effectively than spatial attention masking alone.

The three source directions play different roles here. Occlusion in dense crowds is the application regime and the source of failure cases. Multi-scale detection supplies the architecture, since the decoders in question read multi-scale deformable features, and it also supplies a stratification axis, since we analyze collapse by object size. Over-smoothing supplies both the diagnosis and the repair.

Relation to Prior Work

The closest work is DETR for Pedestrian Detection (PED). PED adapts Deformable DETR to crowds with three ingredients: dense queries, a "rectified attention field" that restricts each layer's cross-attention to the previous layer's predicted box, and supervision of early layers on visible-body boxes. PED constrains where queries look in the image. It never examines whether queries become similar to each other, and its query-to-query self-attention remains a plain softmax average. Our intervention acts on that self-attention, which PED leaves untouched, so the two are complementary and we stack ours on top of PED.

Two papers supply the repair. Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis proves that softmax self-attention is a low-pass filter whose effect residual connections only slow. That paper also proposes AttnScale, which splits attention into low- and high-pass parts and rescales the high-pass part with a learned scalar. It was tested only on ImageNet classification, where the high-pass boost applies uniformly to all tokens. Complete the Missing Half (FB-GNN) adds an explicit node-minus-neighbors "diversification" channel to graph convolution. It reports roughly 10% accuracy gains on heterophilic graphs, where connected nodes have different labels.

A third paper supplies the gate. When Do GNNs Work (ALaGCN) shows that the right amount of aggregation varies per node and gates it with neighborhood statistics. Our gate borrows that idea, but its input is a detection-specific signal: predicted-box overlap.

Overlapping pedestrians are a natural instance of heterophily, since neighboring queries carry different identities. None of these graph methods has been tested on detection queries.

Two works constrain the design. Are we measuring oversmoothing in graph neural networks correctly? warns that global distance-to-mean metrics confuse collapse with legitimate clustering. We therefore measure similarity only between queries that should be distinct. Backward Oversmoothing argues that gradients also homogenize, so we track gradient similarity as well.

PSC-Net uses graph convolution over body parts for occluded pedestrians but never studies smoothing. GraphFPN builds cross-scale graph pyramids, also without studying smoothing. We considered graph pyramids as the main target. We rejected them because they require costly superpixel preprocessing (about 0.12 s per image on CPU), and because the decoder-query setting isolates the occlusion mechanism more cleanly.

Research Program

The first stage is diagnostic. We train Deformable DETR and a PED reimplementation on CrowdHuman, whose annotations include visible-body, full-body and head boxes (CrowdHuman). At each decoder layer $l$ we compute an occluded-pair similarity

$$S_l = \frac{1}{|P|}\sum_{(i,j)\in P}\cos\big(q_i^{(l)}, q_j^{(l)}\big),$$

where $q_i^{(l)}$ is the feature of query $i$ after layer $l$. Here $P$ is the set of query pairs that the final matching step assigns to two different ground-truth people whose full-body boxes overlap with IoU above 0.5. As a control we compute $S_l$ for pairs matched to non-overlapping people. We stratify by box height to expose scale effects. We also test whether high $S_l$ at a pair predicts that one member of the pair is missed or that the pair yields a merged box.

The hypothesis fails if $S_l$ for occluded pairs does not rise with depth relative to the control, or if it does not predict errors. In that case we report the negative result and stop. Such a result would still tell the community that decoder smoothing is not the crowd bottleneck.

The second stage is the intervention. Let $A$ be the decoder's query self-attention matrix and $Q$ the matrix of query features. We replace the self-attention output for query $i$ with

$$\tilde q_i = (AQ)_i + g_i\,\beta\,\big(q_i - (AQ)_i\big).$$

The learned scalar $\beta$ scales the high-pass term $q_i - (AQ)_i$, which is the part of a query that differs from its attention-weighted neighbors. The gate $g_i \in [0,1]$ is a small learned function of the maximum IoU between query $i$'s predicted box from layer $l-1$ and the boxes of other confident queries, together with the entropy of $i$'s attention row. When $g_i = 0$ the layer is ordinary attention. When $g_i = 1$ with a single global $\beta$, the update reduces to an AttnScale-style uniform high-pass boost. That nesting makes the key ablation clean.

The added cost is one small layer that computes the gate and an IoU computation among queries that PED's restricted matching step already performs. The third stage tests transfer to tightly packed small vehicles in DOTA (DOTA). We treat it as exploratory, because no DETR-style crowded-vehicle baseline appears in our sources.

Baselines

The naive reference point is Deformable DETR trained on CrowdHuman under the same schedule. PED reports it at 86.74% AP and 53.98% MR⁻², where MR⁻² is the log-average miss rate and lower is better (DETR for Pedestrian Detection). It tells us how much of any gain is simply crowd adaptation.

The strongest existing method is PED itself, reported at 89.54% AP and 45.57% MR⁻² on CrowdHuman and 47.70% MR⁻² on the CityPersons heavy-occlusion subset. It is the right competitor because it is the only method in our sources that already targets query interference in crowds, through its rectified attention field. We will reimplement it, report our reproduced numbers next to the published ones, and add our layer to the reproduced model. Both models will use the same backbone, the same epochs, the same number of queries and an equal hyperparameter search budget. Beating PED means a consistent MR⁻² reduction on CrowdHuman and on the CityPersons heavy-occlusion subset across three seeds, with no AP loss. We will report gains as differences from our reproduction rather than from the published figure.

The ablation that isolates the key ingredient is the same model with $g_i \equiv 1$, which is a uniform AttnScale-style high-pass channel (Anti-Oversmoothing in Deep Vision Transformers). A second ablation uses a gate driven by random or shuffled IoU signals, which tests whether any extra capacity would help. A classical reference point is a tuned two-stage detector with feature pyramids in the style of the adapted Faster R-CNN from CityPersons. That paper shows simple tuning beating bespoke designs, though the CrowdHuman paper reports roughly 50% miss rate for its FPN baseline in crowds.

Why not just…?

Why not just apply AttnScale uniformly to the decoder? This is the real threat. If a single learned high-pass scale does as well as overlap gating, then our occlusion-specific contribution disappears, and the finding becomes "AttnScale transfers to detection decoders." That would still be useful but much smaller. The $g_i \equiv 1$ ablation settles the question directly. We additionally expect a uniform boost to hurt isolated large objects, whose queries benefit from shared context. Results stratified by overlap and by size will show whether that cost exists.

Why not just add a repulsion or contrastive loss that pushes overlapping queries apart? A loss is simpler and costs nothing at inference, so we include it as a baseline at matched tuning budget. The difference is that a loss acts only on final outputs, whereas collapse, if present, builds up layer by layer and in gradients (Backward Oversmoothing). Our per-layer $S_l$ curves will show whether a loss reverses intermediate collapse or only cosmetically separates the last layer. If the loss matches our method on MR⁻², we will report that and recommend the loss.

Why not just rely on PED's rectified attention field or more queries? Restricting cross-attention controls which pixels a query reads. It does nothing about the self-attention averaging among queries, and two adjacent people's boxes overlap anyway, so the restricted fields overlap too. Measuring $S_l$ on PED versus Deformable DETR tells us how much collapse PED already removes. Adding more queries increases the number of near-duplicate neighbors, which should if anything worsen smoothing. We test this by varying query count.

Why not just pre-train on more crowd data? The CrowdHuman paper shows that pre-training sharply lowers miss rates on CityPersons and Caltech, but our experiments train on CrowdHuman itself, where the crowd problem persists. We will also run the CityPersons evaluation with CrowdHuman pre-training for both PED and our model. If the gap closes under pre-training, our method matters only in the low-data regime, and we will say so.

Proposed Experiments

The decisive sequence has three steps. First, we establish the diagnosis: occluded-pair similarity $S_l$ should rise with depth relative to the non-overlapping control, more steeply for small people, and high values should predict misses on CrowdHuman. Second, we test whether the gated layer flattens that curve and lowers MR⁻² on CrowdHuman and on CityPersons (CityPersons), beyond both reproduced PED and the uniform high-pass ablation. Third, we check that the gains concentrate where the mechanism predicts. That means the improvement should appear on pairs with IoU above 0.5 and on the heavy-occlusion subset, with neutral results on isolated people.

We report AP, MR⁻², and the Jaccard-index-based mMR where the benchmark uses it, along with per-layer $S_l$, gradient cosine similarity, latency and memory. A gain that appears uniformly across occlusion levels, or a gated model indistinguishable from the uniform one, would falsify the occlusion-specific claim even if overall numbers improve. The DOTA small-vehicle transfer is reported as exploratory. The only reference number in our sources for DOTA is the oriented Faster R-CNN at 54.13% mAP, and no reported result was found for a crowded-query DETR on DOTA, so we would establish that baseline ourselves.