Eliminating Predefined Anchors in Object Detection

35 concepts39 papers1 proposal

For years, the dominant paradigm in object detection and tracking relied on a structural scaffold known as anchor boxes. To detect objects of various sizes and shapes, researchers tiled an image with thousands of predefined bounding boxes acting as geometric templates. The neural network's job was not to find an object from scratch, but to classify these anchors and regress slight adjustments to their coordinates.

While highly successful, this dense-to-sparse template matching strategy introduces a severe hyperparameter bottleneck. It forces engineers to manually configure the scales, aspect ratios, and quantities of anchors before training even begins. Over the past few years, a powerful shift has occurred: researchers are abandoning hand-crafted anchors in favor of anchor-free, point-based, and learnable proposal representations.

The Fragility of Hand-Crafted Anchors

Why move away from anchors? The core issue is that manual anchor configuration creates extreme hyperparameter sensitivity and limits a model's ability to generalize.

When designing an anchor-based detector, an engineer must make rigid assumptions about the dataset's geometry. If an object presents an unexpected aspect ratio or scale, the predefined anchors will fail to overlap with it sufficiently during training. As noted by Michael Baumgartner et al. in nnDetection, this manual configuration pipeline creates a significant bottleneck, often driving researchers to use semantic segmentation even when object detection is the more appropriate clinical or practical objective.

Furthermore, tiling an image with thousands of anchors creates a massive class imbalance problem—the vast majority of anchors cover empty background. To train the network, researchers rely on static, heuristic rules like Intersection-over-Union (IoU) thresholds to assign anchors to ground-truth objects. As Yang Liu et al. demonstrate in MogFace, these rigid assignment strategies fail to identify high-quality samples for difficult or unconventional instances. This mismatch forces complex, sensitive tuning of assignment thresholds, as explored by Xu Tang et al. in HAMBox.

Object Detection as Keypoint Estimation

The most intuitive alternative to drawing boxes is to predict points. Instead of asking a network to choose from a discrete set of box templates, point-based methods reframe object detection as a continuous keypoint estimation problem.

Hei Law et al. pioneered this shift with CornerNet, which detects an object as a pair of keypoints: its top-left and bottom-right corners. By predicting continuous corner heatmaps and grouping them via associative embeddings, the model bypasses anchor boxes entirely. To resolve the geometric ambiguity of matching independent corners in dense scenes, Zhiwei Dong et al. introduced CentripetalNet. This approach equips corners with "centripetal shifts"—2D vectors pointing toward the object's center—allowing the model to cleanly pair corners that share a geometric center.

Anchor-Based Approach:
[ Tile thousands of fixed boxes ] ──> [ Regress offsets from closest template ]

Point-Based Approach:
[ Predict continuous keypoints ]  ──> [ Group points geometrically ]

Alternatively, models can localize objects by finding their center points. Ali Etemad et al. proposed ObjectBox, which treats object centers as the primary anchor-free primitive, regressing the distances to the object boundaries directly from the center pixel. Similarly, Hao Chen et al. in Detection of Multiscale Center Point Objects Based on Parallel Network decoupled localization from attribute regression using center-point heatmaps.

This point-based formulation scales naturally across domains. For instance, in spatio-temporal action detection, Yixuan Li et al. developed the MovingCenter Detector in Actions as Moving Points, tracking human actions as trajectories of moving points rather than complex, rigid 3D bounding cuboids.

Dense Per-Pixel and Pillar Predictions

Another powerful anchor-free strategy is to map features directly to bounding box coordinates at every spatial location, treating detection as a dense, fully convolutional regression task.

In visual tracking, traditional Siamese networks were heavily constrained by anchor configurations that struggled with severe object deformation. To solve this, Dongyan Guo et al. introduced SiamCAR, which decomposes tracking into per-pixel classification and direct bounding box regression. This line of work, extended by Ying Cui et al. in Joint Classification and Regression for Visual Tracking with Fully Convolutional Siamese Networks, eliminates the need to tune scale and aspect ratio hyperparameters. Similar per-pixel and box-adaptive paradigms were introduced by Zedu Chen et al. in Siamese Box Adaptive Network for Visual Tracking and Fan Tang et al. in SiamCPN.

This dense regression philosophy also simplifies complex 3D and oriented detection tasks:

  • 3D Point Clouds: Yue Wang et al. developed an anchor-free framework in Pillar-Based Object Detection for Autonomous Driving, substituting hand-crafted 3D bounding box priors with a per-pillar prediction strategy.
  • Aerial Imagery: Objects in aerial views appear at arbitrary angles, meaning anchor-based models must tile boxes across multiple orientations, causing massive computational overhead. Licheng Jiao et al. bypassed this in FCOSR by using an anchor-free rotated detector that directly predicts oriented bounding boxes from feature maps.

Learnable Proposals and Generative Denoising

The latest evolution in removing manual priors moves away from dense spatial grids entirely, replacing them with a small set of learnable proposals or generative denoising processes.

Instead of evaluating thousands of candidate locations, Peize Sun et al. proposed Sparse R-CNN. This architecture utilizes a tiny, fixed set of learnable proposal boxes and proposal features that act as dynamic queries. These queries interact with image features through an attention mechanism, iteratively refining their coordinates across network stages. This completely eliminates both dense anchor grids and Non-Maximum Suppression (NMS) post-processing. Jianhong Han successfully extended this sparse paradigm to 3D object detection in SparseDet.

Taking the idea of dynamic proposals to its logical extreme, Xin Zhou et al. introduced Diffusion-based 3D Object Detection with Random Boxes. Rather than using any geometric priors or learnable templates, the model initializes detection with completely random 3D boxes and treats object detection as a generative denoising task, progressively refining chaos into precise bounding boxes.

Why It Works: The Intuition

The fundamental reason anchor-free and learnable proposal methods outperform traditional anchor-based pipelines comes down to optimization alignment.

In an anchor-based system, the network is forced to learn a highly non-linear mapping from an arbitrary, human-defined template box to the actual object boundaries. If the anchor is poorly matched to the object, the gradient updates become highly unstable because the network has to compensate for a bad initial guess.

By switching to anchor-free representations, we align the network's architecture with the continuous, intrinsic geometry of the data. When a network predicts a corner, a center point, or refines a sparse query, it optimizes coordinates directly in a continuous space.

Crucially, this does not mean losing the mathematical benefits of box-level supervision. For instance, Jiachen Li demonstrates in Pseudo-IoU that an anchor-free model can dynamically generate virtual "pseudo boxes" during training to compute IoU scores for high-quality label assignment, without incurring any of the inference-time overhead or rigidity of traditional anchor grids.

By shifting the burden of geometric modeling from manual human heuristics to continuous neural optimization, modern detectors achieve greater flexibility, cleaner architectures, and superior generalization across diverse visual domains.

Key Foundational Work

MetaAnchor: Learning to Detect Objects with Customized Anchors (2018) — contributed 1 concept elements to this direction.
CornerNet: Detecting Objects as Paired Keypoints (2018) — contributed 1 concept elements to this direction.
High-Level Semantic Feature Detection: A New Perspective for Pedestrian Detection. (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_manual_anchor_configuration_7a7d211f,
  title = {manual anchor configuration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-anchor-configuration/rdr_7a7d211f438c4b65929405dc63a896fc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_manual_anchor_configuration_7a7d211f_v1,
  title = {manual anchor configuration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-anchor-configuration/rdr_7a7d211f438c4b65929405dc63a896fc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake