Towards Learning Monocular 3D Object Localization From 2D Labels using the Physical Laws of Motion

Daniel Kienzle, Julian Lorenz, Katja Ludwig, Rainer LienhartarXiv 2023 · arXiv

TL;DR

Researchers have developed a way to train neural networks for precise 3D object localization using only a single camera and easy-to-obtain 2D labels. By incorporating differentiable physical laws (like gravity and bouncing) into the training loop, the model learns to infer the "latent" third dimension, achieving a mean error of just 6 cm on real-world data without ever seeing a 3D label during training.

What Problem Does This Paper Address?

Determining the 3D position of an object (like a ball in sports) from a single 2D image is a classic "ill-posed" problem—mathematically, there are infinite 3D points that could project to the same 2D pixel. Traditional solutions usually require expensive multi-camera setups (like Hawk-Eye) or massive datasets of 3D ground-truth labels, which are difficult and costly to collect for amateur sports or niche environments.

The authors aim to bridge this gap by asking: Can we use our knowledge of how the world works—specifically the laws of physics—to teach a model about depth using only 2D videos?

Methods

The proposed framework consists of two main components: the Position Estimation Network (PEN) and the Physics Aware Forecast Module (PAF).

  1. Position Estimation Network (PEN): This is the part used during inference. It takes a single image and predicts two things: a 2D heatmap (where the ball is in the image) and a depthmap (how far away the ball is). PEN architecture showing the generation of heatmap and depthmap
Figure 3. Position Estimation Network (PEN) architecture for generating both heatmap (HM) and depthmap (DM) from an input image. Features from the backbone network are concatenated and passed through the two heads. One generates a heatmap and the other produces a depthmap.
  1. Physics Aware Forecast Module (PAF): This module is the "teacher" used only during training. It uses Neural Ordinary Differential Equations (NDEs) to solve the equations of motion. By looking at the ball's predicted 3D position across three consecutive frames ($t_{n-1}, t_n, t_{n+1}$), it estimates the ball's velocity. It then uses physics (gravity, air resistance, and floor/wall collisions) to forecast where the ball should be at a future time ($t_{n+\Delta n}$).

The "Aha!" Moment in Training: The model is supervised by two losses. The first is a standard 2D heatmap loss. The second, more novel loss, takes the PAF's predicted future 3D position, projects it back into a 2D image coordinate, and compares it to the PEN's prediction for that future frame. To minimize this "future loss," the network is forced to predict depths that are physically consistent with the observed 2D motion.

Figure 2. Training and inference pipeline. The Position Estimation Network (PEN) estimates the heatmap (HM) and depthmap (DM) for each input image, from which the 2D image coordinates and camera depth are extracted. Using the intrinsic camera matrix, we compute the 3D camera coordinates from the image coordinates and camera depth. With the extrinsic camera matrix, the world coordinates ⃗r are calculated. During training, we apply the PEN to multiple coherent frames for the estimation of the velocity ⃗v. The Physics Aware Forecast Module (PAF) utilizes these initial conditions to calculate the world coordinates at a later time tn+∆n by solving the differential equations of motion. These coordinates are then projected back to image coordinates and the future loss is calculated. The heatmap loss is calculated by comparing the predicted and ground truth heatmaps at all time steps. During inference, the PEN is applied to single images. Thus, it is robust to incalculable situations like sudden wind or human interventions. Blue arrows indicate the use of camera matrices, and dotted red lines indicate detached gradients.

Datasets

The authors evaluated their method on three distinct types of data:

  • Synthetic Dataset (SD): Created using the MuJoCo physics engine. It includes three environments with varying obstacles and nine different camera angles.
  • Real Dataset (RD): Monocular videos of a rubber ball bouncing in a room, recorded with a ZED 2i stereo camera (the stereo data was used only to generate ground truth for evaluation, not for training).
  • Spring Dataset: A specialized synthetic dataset where a ball is attached to a spring, demonstrating that the method generalizes to different physical systems beyond simple gravity.
Example images from synthetic and real datasets

Key Insights and Ideas

  • Physics as a Differentiable Supervisor: By implementing the Hamilton formalism within a differentiable framework (NDEs), the laws of motion act as a geometric constraint that backpropagates errors into the depth estimation head.
  • Inference Flexibility: Even though the model is trained on sequences to understand motion, it performs inference on single images. This makes it robust to non-deterministic events (like a player hitting the ball or a sudden gust of wind) that would break a purely physics-based trajectory-fitting model.
  • Emergent Environmental Geometry: Interestingly, the model learns a full depthmap of the scene (floor, walls, tables) even though it is only explicitly supervised at the ball's location. The network learns that the ball's depth is related to the surrounding context.
  • Saliency: Analysis shows the network primarily looks at the ball's size to determine depth but also considers the local surroundings, suggesting it uses more than just simple diameter measurements.

Key Results

  • High Precision: On the real-world dataset, the model achieved a mean distance error (DtG) of 6–7 cm.
  • Competitive with Supervised Learning: In ablation studies, the physics-based "2D-predict" method performed significantly better than using noisy 2D ground truth labels directly and was surprisingly close to a baseline trained with full 3D supervision.
  • Generalization: The model successfully generalized to unseen camera locations and different environmental geometries in synthetic tests.

Limitations and Future Work

  • Resolution Constraints: The current experiments use relatively low-resolution images (e.g., 224x224). In professional sports broadcasting, the ball is often just a few pixels wide in a high-res frame, which might require an initial object detection crop before localization.
  • Physics Modeling: The method requires a known environment (e.g., floor height and wall positions) to model collisions accurately.
  • Complex Dynamics: While the authors tested gravity and springs, more complex effects like ball spin (Magnus effect) were not included in the current physics module, though the authors note that differentiable engines like BRAX could be integrated to handle this.

Future research will focus on scaling the method to higher resolutions and improving the generalization ability to entirely unknown environments.

Summary updated

Abstract

We present a novel method for precise 3D object localization in single images from a single calibrated camera using only 2D labels. No expensive 3D labels are needed. Thus, instead of using 3D labels, our model is trained with easy-to-annotate 2D labels along with the physical knowledge of the object's motion. Given this information, the model can infer the latent third dimension, even though it has never seen this information during training. Our method is evaluated on both synthetic and real-world datasets, and we are able to achieve a mean distance error of just 6 cm in our experiments on real data. The results indicate the method's potential as a step towards learning 3D object location estimation, where collecting 3D data for training is not feasible.
BibTeX
@misc{kienzle2023towards,
  author = {Daniel Kienzle and Julian Lorenz and Katja Ludwig and Rainer Lienhart},
  title = {{Towards Learning Monocular 3D Object Localization From 2D Labels using the Physical Laws of Motion}},
  year = {2023},
  eprint = {2310.17462},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.17462},
}
Figure