Center-point detectors find each object as a peak in a heatmap and read its box size from the single feature vector at that peak. This makes them fast and removes the duplicate-suppression step. It also makes them fragile: when the center of a person is hidden behind another person, the box is regressed from features that belong to someone else. We propose a small second step. Around each detected center, the detector gathers features from a grid inside its first-guess box. It then fits a low-dimensional linear subspace to them with a robust, absolute-distance criterion and anchors that fit at the center feature. Features from the object itself, such as its head, torso and legs, vary along a few directions and land near the subspace. Features from occluders and neighbors land far from it and are down-weighted automatically. The box is then re-regressed and rescored from the down-weighted features alone.
The bet is that the right unit of context for an occluded object is "every location that is consistent with this instance." That is wider than the center pixel and narrower than the whole image or a whole row and column. A robust subspace fit, rather than a learned attention map, is a principled way to decide that consistency without extra labels. A side effect gives a sharp test. The per-location weights from the fit form a soft visibility mask learned with no visibility supervision, and CrowdHuman supplies visible-region boxes to check it against.
Core Claim
Replacing center-pixel box regression with regression from features weighted by a per-instance robust subspace fit should do three things. It should reduce misses and box errors for occluded instances in crowded scenes. It should do so by more than a learned attention or region-pooling head of matched size. And it should only work when the fitted subspace has rank above zero.
Each part of this claim can fail on its own. A plain robust average (rank zero) treats an object's legitimate parts, like head versus torso, as outliers of one another. A plain squared-error fit gets dragged toward whichever occluder is largest. If a small learned head matches the robust fit, the subspace story is unnecessary.
Where This Sits
Four lines of prior work shape the design.
The first motivates the problem. Objects as Points regresses all box attributes from the peak pixel and replaces duplicate suppression with a 3×3 max-pool. We keep both choices and add a refinement after the peak. CenterNet: Keypoint Triplets for Object Detection showed that center quality is the main ceiling: AP on MS-COCO would rise from 47.0% to 56.5% with perfect centers. Its Center Pooling and Cascade Corner Pooling, extended to multiple resolutions in CenterNet++ for Object Detection, gather context by taking the maximum response along entire rows and columns. In a crowd, that maximum can come from a neighbor. We replace the maximum over an axis with a consistency test over a 2D region.
The second line constrains the design. RPCANet++ unrolls robust PCA to split an image into a low-rank background and a sparse target. Its authors report that it breaks down once targets exceed about 15% of the image. Complete Moving Object Detection in the Context of Robust Subspace Learning learns an object subspace only implicitly inside a GAN, and only for video background subtraction. Both treat the object as the sparse anomaly. We invert the roles: the object is the inlier set of a per-instance fit, and clutter is the outlier. This means large objects are not a problem. Neither paper outputs boxes, classes or separate instances.
The third line supplies the method. An Overview of Robust Subspace Recovery separates least-absolute-deviation subspace fitting (minimizing summed distances) from variance maximization. We use the former. Iteratively Reweighted Least Squares Algorithms for L1-Norm Principal Component Analysis gives reweighting solvers, including a variant that avoids a full decomposition at each step and reaches near-identical error in about 20% of the time. That makes a few unrolled, differentiable iterations per detection affordable. Dual Principal Component Pursuit tolerates very high outlier fractions but is designed for subspaces of high relative dimension. Ours are of very low dimension inside a 64-dimensional embedding, so we use it only as a comparison solver. Rethinking PCA Through Duality shows that an absolute-distance robust PCA beats squared-error PCA under 15% outliers on MNIST (87% vs 76–80% accuracy). That is evidence the robust criterion matters, though only on flat vectors.
Two open problems from this literature turn out to be easier in the detection setting. Choosing the subspace dimension becomes a small sweep we report. Centering an affine subspace robustly is handled by the detector itself, because the predicted center feature supplies the offset.
Research Program
We start from the CenterNet design of Objects as Points and add one head: a 64-dimensional per-pixel instance embedding. It is trained with a pull/push loss so that pixels of the same instance cluster together and pixels of different instances separate, including instances of the same class.
The embedding is needed because a person-appearance subspace would otherwise accept a neighboring person's torso as an inlier. This is the most important design risk in the method. We report results with and without the embedding.
For each of the top 100 peaks, the detector proceeds in three steps. It samples a 7×7 grid of embedding vectors $f_i$ inside the first-pass box, enlarged by 20%. It then solves
$$\min_{U^\top U = I_d} \sum_{i} \left\| (I - UU^\top)(f_i - f_c) \right\|_2 ,$$
where $f_c$ is the embedding at the detected center and $U$ is a basis for a $d$-dimensional subspace with $d$ between 1 and 4. It runs $T=5$ unrolled reweighting iterations, initialized from a squared-error fit. Finally, it sets each location's weight to $w_i = \exp(-r_i/\tau)$, where $r_i$ is that location's residual distance to the fitted subspace and $\tau$ is learned.
The original backbone features, weighted by $w_i$, are pooled and passed to a small head. That head outputs a box correction and a rescored confidence. The added cost is a few small matrix products per detection, which we will measure as frames per second.
The analyses that follow are designed to break the claim, not to tune it:
- A rank sweep from $d=0$, a robust median of the embeddings, up to $d=8$.
- A swap of the fitting criterion between absolute distance and squared error at fixed rank.
- Gains stratified by occlusion level, using CrowdHuman's ratio of visible box to full box.
- A comparison of the weight maps $w$ against the visible-region annotations.
Baselines
Naive reference. The do-nothing reference is the unmodified center-point detector of Objects as Points, retrained on each target dataset with the same backbone, schedule and augmentation as our model. On COCO it reports 28.1% AP with ResNet-18 at 142 FPS and 45.1% AP with Hourglass-104. Our COCO runs must reproduce these numbers within about half a point before any comparison is trusted.
Strongest existing anchor-free method. The strongest existing method for anchor-free context aggregation is CenterNet++ for Object Detection. It reports 57.1% AP on COCO with Swin-L and 43.6% AP at 30.5 FPS for its real-time variant. It is the direct competitor because its pooling targets exactly the center bottleneck we target. We will run its public configuration on CrowdHuman and COCO-C with the same backbone as ours (Hourglass-104, the setting with reported triplet numbers), the same epochs, and the same number of hyperparameter trials. We found no reported result on Lacuna for any center-point detector on CrowdHuman. The only crowd reference in the file is the roughly 50% miss rate of a feature-pyramid two-stage detector reported in CrowdHuman. Beating CenterNet++ means a lower log-average miss rate on CrowdHuman full-body boxes, with non-overlapping confidence intervals across three seeds, at no worse than equal COCO AP.
Learned-refinement ablation. This ablation isolates the key ingredient. It uses the same sampled grid, the same embedding, and the same final head. The robust fit is replaced with a learned attention pooling layer whose parameter count and FLOPs match or exceed ours. If this ablation matches us on CrowdHuman and on the visibility test, the robust subspace is not doing the work.
Inner ablations. These swap only the fitting step: rank-zero robust median, squared-error PCA at the same rank, and the dual pursuit solver from Dual Principal Component Pursuit.
Corruption robustness. As the robustness reference, we include the stylized-training detector of Benchmarking Robustness in Object Detection. It raised retained performance under corruption from 60.4% to 69.9% on PASCAL-C and from 33.4% to 47.4% on Cityscapes-C. We will also combine stylized training with our module, to test whether the two help in different ways.
Why not just…?
Why not just use a Swin or ViT backbone for global context? CenterNet++ already uses Swin-L and still relies on hand-designed center and corner pooling to suppress false centers. Global attention also does not tell the box head which pixels belong to this instance rather than its neighbor. Our module sits on top of any backbone, so the experiment is direct: we run with Swin-T as well. If the gain disappears with an attention backbone, we will report that the method only fixes a convolutional weakness.
Why not just train with stylization or corruption augmentation? Augmentation targets texture shift. It does not target structural occlusion, and Benchmarking Robustness in Object Detection shows that stylized detectors still lose a large share of clean performance. The occlusion-stratified CrowdHuman analysis is where augmentation should help least and our module most. The combined run shows whether the two effects add up. On COCO-C, augmentation may well beat us, and we will say so if it does.
Why not just use squared-error PCA or a median? This is the cheapest ablation and lives inside our own model. The duality paper's MNIST gap (87% vs 76–80%) suggests squared error will be pulled by occluders. A median, on the other hand, cannot represent the spread of an object's parts. If the rank-zero median matches rank 2–4, the subspace claim fails, and we would report a simpler robust-pooling result instead.
Why not just add a learned second-stage refinement head, as two-stage detectors do? This is the real threat. A learned head on the same sampled features can, in principle, learn to ignore occluders. Our argument is that the robust fit performs this rejection per instance, adapts to how much of the region is occluder, and needs no visibility labels. A learned head must infer all of this from data. The matched-capacity ablation settles the question. So does the visibility test, which asks whether a learned attention map also aligns with the visible regions.
Proposed Experiments
The primary benchmark is CrowdHuman, which averages 22.6 persons per image and 2.4 heavily overlapping pairs per image. We use its 15,000 training and 4,370 validation images. We report log-average miss rate for full-body boxes and break it down by visible-to-full area ratio.
The decisive check is that the gain over the naive and learned-refinement baselines should grow monotonically with occlusion level. A second decisive check uses the visibility weights. We threshold $w$ and measure IoU between the resulting mask and the annotated visible box. The weights must align with visible regions clearly better than the learned attention map does, even though neither was trained on visibility.
On MS-COCO we require AP no lower than the reproduced baselines. This shows the module does not hurt uncrowded scenes, and we also report the FPS cost. On COCO-C and Cityscapes-C we report mean and retained performance under corruption across 15 corruption types and 5 severities. This is a secondary test of whether residual-based down-weighting also rejects corrupted locations.
The rank sweep, the swap of fitting criterion, and the runs with and without the instance embedding are reported on CrowdHuman validation. One result would refute the central claim even if overall numbers improve: rank zero performing as well as rank above zero, or squared error performing as well as absolute distance.
References
- Objects as Points
- CenterNet: Keypoint Triplets for Object Detection
- CenterNet++ for Object Detection
- RPCANet++: Deep Interpretable Robust PCA for Sparse Object Segmentation
- Complete Moving Object Detection in the Context of Robust Subspace Learning
- An Overview of Robust Subspace Recovery
- Iteratively Reweighted Least Squares Algorithms for L1-Norm Principal Component Analysis
- Dual Principal Component Pursuit for Robust Subspace Learning: Theory and Algorithms for a Holistic Approach
- Rethinking PCA Through Duality
- CrowdHuman: A Benchmark for Detecting Human in a Crowd
- Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming