When designing deep learning models for spatial tasks—such as 3D human pose estimation, object detection, or surface reconstruction—the most intuitive approach is often to map the input data directly to absolute spatial coordinates. If you want to know where a joint, a bounding box corner, or a surface point is, why not simply train a neural network to output its exact $(x, y, z)$ position?
Despite its simplicity, this "holistic" or direct regression strategy frequently fails in practice. Neural networks tasked with direct coordinate regression suffer from slow convergence, high sensitivity to noise, and poor generalization to unseen environments.
The core issue is not a lack of model capacity, but rather an optimization bottleneck. Direct regression forces networks to map highly non-linear visual features to an unbounded, high-variance continuous space. To build robust spatial models, researchers have shifted toward architectures that decompose absolute coordinates into localized, relative, or bounded geometric primitives.
Why Direct Regression Fails
To understand why direct coordinate regression is problematic, we must look at the optimization landscape it creates. When a network attempts to predict absolute coordinates across a global scene, it faces three distinct mathematical hurdles:
- High Target Variance: In a large or unconstrained scene, the absolute coordinates of an object can span a massive range of values. As noted by Danfei Xu et al. in their work on PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation, this vast search space creates high variance in the regression targets, making it difficult for standard loss functions to provide stable gradient updates.
- Unbounded Output Spaces: Unlike classification tasks where outputs are constrained to a bounded probability distribution (like a softmax layer), raw coordinate offsets are inherently unbounded. Liuhao Ge et al. observe that this lack of normalization makes training highly unstable, as early training iterations can produce wild, erratic predictions far from the true data manifold.
- Complex Non-Linear Mappings: Transforming a flat 2D pixel grid into a 3D coordinate space requires the network to implicitly learn complex perspective geometry. Fuyang Huang et al. demonstrate in Structure-Aware 3D Hourglass Network for Hand Pose Estimation from Single Depth Image that this highly non-linear mapping lacks spatial stability, causing models to struggle with complex structural constraints.
When these factors combine, the network is forced to solve two problems simultaneously: identifying what or where an object is contextually, and calculating its exact numerical distance. This dual burden often leads to geometric artifacts, such as outliers or surface distortion.
Strategy 1: Localized Anchors and Residuals
The most prominent solution to coordinate instability is to replace absolute global targets with localized, relative targets. Instead of asking a network to guess where an object sits in the entire world, the problem is reframed: given a nearby reference point (an anchor), what is the small correction needed to reach the target?
Global Coordinate Space
[0, 0, 0] ---------------------------------------------> Target: [145.2, 82.1, 210.4] (High Variance)
Localized Anchor Space
[0, 0, 0] --> Anchor: [145.0, 82.0, 210.0] --> Residual: [+0.2, +0.1, +0.4] (Low Variance)
In PointFusion, Danfei Xu et al. utilize raw 3D points from a point cloud as spatial anchors. By predicting bounding box coordinates relative to these local points rather than the global origin, the regression targets become tightly bounded, dramatically stabilizing training.
A similar principle applies to category-level object estimation. In RBP-Pose: Residual Bounding Box Projection for Category-Level Pose Estimation, Ruida Zhang et al. leverage a generic shape prior as an anchor. The network is trained to predict small, zero-mean "residual vectors" that bridge the gap between the generic prior and the specific object instance. Because these residuals operate in a well-behaved, low-variance numerical range, the optimization landscape becomes significantly smoother.
This strategy is also highly effective for articulated structures like the human body. As shown by Ángel Martínez-González et al. in Residual Pose: A Decoupled Approach for Depth-based 3D Human Pose Estimation, learning localized refinements or corrections from a rough initial estimate prevents the network from getting trapped in local minima—a common failure mode when feed-forward networks attempt to map raw pixels directly to absolute 3D joints.
Strategy 2: Coarse-to-Fine Classification (Bucketing)
Another powerful alternative to direct regression is to discretize the continuous coordinate space. By converting a continuous regression problem into a discrete classification problem, models can exploit the inherent stability of cross-entropy losses before refining the final output.
In Hierarchical Scene Coordinate Classification and Regression for Visual Localization, Xiaotian Li et al. tackle the problem of predicting 3D scene coordinates across expansive environments. Directly regressing coordinates in a massive world space causes slow convergence. The authors resolve this by first classifying pixels into discrete, localized regions (coarse localization) and then regressing the exact relative offset within that specific region (fine localization).
Continuous Space: |-----------------------------------------------------------| (Hard to regress)
Discretized Boxes: [ Box A ] [ Box B ] [ Box C ] [ Box D ] [ Box E ] [ Box F ] (Easy to classify)
Fine Regression: [--|--] (Small, stable offset)
This "divide and conquer" approach is equally potent in 2D object detection. Jiaqi Wang et al. introduce this concept to bounding box localization in Side-Aware Boundary Localization for More Precise Object Detection. When anchor boxes are far from the target object, direct offset regression fails due to high target variance. By implementing a two-step bucketing scheme that first isolates a coarse boundary region via classification and then pinpoints the exact edge, the authors bypass the instability of long-range regression entirely.
Strategy 3: Relative and Ordinal Relationships
When predicting complex structures, humans rarely calculate absolute millimeter distances from scratch; instead, we rely on relative spatial arrangements and structural constraints. Emulating this cognitive shortcut within neural networks provides a highly stable alternative to coordinate regression.
In DRPose3D: Depth Ranking in 3D Human Pose Estimation, Min Wang et al. bypass absolute 3D coordinate estimation by focusing on relative depth ranking between joints. By transforming an ambiguous 2D-to-3D mapping into a series of pairwise classification problems (e.g., "is the left wrist closer to the camera than the left elbow?"), the model avoids numerical instability and generalizes far better to unseen camera perspectives.
This concept is expanded to multi-person scenarios by Can Wang et al. in HMOR: Hierarchical Multi-person Ordinal Relations for Monocular Multi-person 3D Pose Estimation. The authors argue that supervising networks purely with absolute coordinates fails to capture the structural context of a scene. By introducing hierarchical ordinal relationships as a supervisory signal, they demonstrate that relative depth and angular constraints are far more resilient to image noise than absolute coordinate targets.
Absolute Targets: Joint A: (120, 45, 250) | Joint B: (125, 80, 290) -> High sensitivity to pixel noise
Relative Targets: Bone Vector: (+5, +35, +40) or Depth Order: A is closer than B -> Structurally stable
Similarly, Xiao Sun et al. show in Compositional Human Pose Regression that regressing the "bones" (the relative vectors connecting joints) rather than individual joint locations inherently stabilizes the learning process. The structural composition of the target itself acts as a regularizer, preventing the network from outputting anatomically impossible coordinates.
Strategy 4: Bounded Fields and Distance Functions
For dense geometric tasks like point cloud upsampling or surface normal estimation, predicting raw 3D vectors directly can lead to severe geometric artifacts, such as outliers or surface shrinkage. A final, highly effective strategy replaces vector regression with the estimation of bounded scalar fields.
In Grad-PU: Arbitrary-Scale Point Cloud Upsampling via Gradient Descent with Learned Distance Functions, Yun He et al. address the problem of generating new points on a 3D surface. Directly predicting the absolute 3D coordinates for new points is a high-dimensional, unstable task. The authors decouple the process: instead of forcing the network to place points directly in 3D space, they train the network to estimate a simple scalar distance to the underlying surface manifold. Once this distance field is learned, points are precisely shifted onto the surface via gradient descent. Estimating a scalar field is significantly less complex for the network than regressing full 3D coordinates, leading to clean, artifact-free geometric boundaries.
Summary
The shift away from direct coordinate regression represents a fundamental realization in spatial deep learning: the mathematical framing of a target matters as much as the network architecture itself. By breaking down absolute, high-variance global coordinates into localized anchors, discrete buckets, relative structural constraints, or bounded scalar fields, modern architectures transform unstable optimization landscapes into highly manageable learning tasks.
Key Foundational Work
Point-to-Point Regression PointNet for 3D Hand Pose Estimation (2018) — contributed 1 concept elements to this direction.
DRPose3D: Depth Ranking in 3D Human Pose Estimation (2018) — contributed 1 concept elements to this direction.
Structure-Aware 3D Hourglass Network for Hand Pose Estimation from Single Depth Image (2018) — contributed 1 concept elements to this direction.