Inverse Neural Rendering for Explainable Multi-Object Tracking

Julian Ost, Tanushree Banerjee, Mario Bijelic, Felix HeidearXiv 2024 · DOI · arXiv

Abstract

Modern 3D multi-object tracking (MOT) from monocular RGB cameras often relies on feed-forward neural networks that struggle with generalization and lack interpretability. This paper introduces a novel approach that recasts 3D MOT as an Inverse Rendering (IR) problem. By optimizing over the latent space of a generative 3D object prior at test time, the proposed method achieves remarkable zero-shot generalization to unseen real-world datasets. The framework provides inherently interpretable tracking results by explaining images through synthesis rather than black-box prediction.

Introduction

3D Multi-Object Tracking (MOT) from a single camera is a notoriously difficult task due to the inherent depth ambiguity in 2D images. While deep learning has significantly advanced the field, current state-of-the-art methods face three critical challenges:

  1. Poor Generalization: Feed-forward models often overfit to specific camera parameters or geographic locations, leading to failure on new datasets.
  2. Lack of Interpretability: Reasoning occurs within high-dimensional feature spaces, making it difficult for humans to audit failures.
  3. Geometric Inconsistency: Standard 2D networks struggle to respect 3D physical constraints like object shape and perspective during inference.

To address these, this work moves away from direct attribute prediction. Instead, it seeks to "explain" the input image by finding the 3D scene parameters that, when rendered, best match the observed pixels.

Methodology: Tracking via Synthesis

The core of the system is a shift from discriminative modeling to generative optimization. The pipeline consists of a generative prior, an inverse rendering loop, and a robust tracking module.

Generative Object Prior

The authors utilize GET3D, a generative model trained exclusively on synthetic data (ShapeNet). This model generates high-quality textured 3D meshes from a disentangled latent space representing shape ($z_S$) and texture ($z_T$). By leveraging a prior trained on synthetic data, the system avoids overfitting to real-world sensor noise.

The Inverse Rendering Pipeline

Instead of a standard forward pass, the system performs test-time optimization. Given an initial 2D or 3D detection, the system initializes a 3D object and iteratively adjusts its position, orientation, shape, and texture latents.

Figure 2: The Inverse Rendering and Tracking Pipeline. The system optimizes latent codes ($z_t, z_s$) through a generative prior to match target predictions, followed by interpretable matching using 3D IoU and latent embeddings.

The optimization minimizes a composite loss function:

  • Pixel-wise $L_2$ Loss ($L_{RGB}$): Ensures raw color alignment.
  • Perceptual Similarity (LPIPS): Matches high-level features of the observed vehicle.

To ensure stability, the system employs an alternating optimization schedule, first solving for coarse appearance before jointly refining the object's shape and 3D pose.

Tracking and Latent Matching

Once objects are inverse-rendered, they are maintained using a Kalman Filter. Data association between frames is performed using a weighted score of 3D Intersection-over-Union (IoU) and the cosine distance between optimized latent embeddings. This latent matching is more robust than raw image feature matching because it compares the underlying 3D properties of the objects.

Evaluation and Results

Zero-Shot Generalization

A key strength of this approach is its ability to perform on real-world data without being trained on it. The generative prior was trained only on synthetic ShapeNet data, yet the model was evaluated on the nuScenes and Waymo Open Dataset (WOD).

On the nuScenes "cars" test split, the method achieved an AMOTA of 0.413, outperforming established baselines like AB3DMOT (0.387) and CenterTrack (0.202) when using the same detection inputs. Crucially, the model maintained this performance across different geographic domains without any fine-tuning.

Qualitative Performance

Qualitative analysis demonstrates the tracker's robustness in diverse and challenging environments, including dense urban traffic and cluttered parking lots.

Figure 3: Qualitative tracking results across diverse real-world scenarios (Suburban, Urban, Clutter, Parking lot) from time $t_0$ to $t_3$. The method successfully maintains tracks across varying lighting and density.

Discussion and Interpretability

Beyond performance metrics, this method offers "interpretability for free." If a track fails, a human operator can inspect the rendered output. Discrepancies between the rendered car and the actual image (e.g., mismatched shadows or incorrect geometry) provide immediate, diagnosable evidence of why the optimization failed.

Limitations and Future Work

The primary bottleneck is computational cost. Iterative optimization currently takes approximately 0.3 seconds per frame per object, which is not yet suitable for real-time applications. Additionally, the method still depends on an external detector for initial object proposals. Future research will focus on accelerating the rendering pipeline and expanding the framework to handle a wider variety of object classes "in-the-wild."

Conclusion

By framing 3D multi-object tracking as an inverse rendering problem, this work demonstrates that generative priors can bridge the gap between synthetic training and real-world application. The resulting system is not only more generalizable than traditional feed-forward networks but also provides a level of transparency essential for safety-critical autonomous systems.

Summary updated

Abstract

Today, most methods for image understanding tasks rely on feed-forward neural networks. While this approach has allowed for empirical accuracy, efficiency, and task adaptation via fine-tuning, it also comes with fundamental disadvantages. Existing networks often struggle to generalize across different datasets, even on the same task. By design, these networks ultimately reason about high-dimensional scene features, which are challenging to analyze. This is true especially when attempting to predict 3D information based on 2D images. We propose to recast 3D multi-object tracking from RGB cameras as an \emph{Inverse Rendering (IR)} problem, by optimizing via a differentiable rendering pipeline over the latent space of pre-trained 3D object representations and retrieve the latents that best represent object instances in a given input image. To this end, we optimize an image loss over generative latent spaces that inherently disentangle shape and appearance properties. We investigate not only an alternate take on tracking but our method also enables examining the generated objects, reasoning about failure situations, and resolving ambiguous cases. We validate the generalization and scaling capabilities of our method by learning the generative prior exclusively from synthetic data and assessing camera-based 3D tracking on the nuScenes and Waymo datasets. Both these datasets are completely unseen to our method and do not require fine-tuning. Videos and code are available at https://light.princeton.edu/inverse-rendering-tracking/.
BibTeX
@misc{ost2024inverse,
  author = {Julian Ost and Tanushree Banerjee and Mario Bijelic and Felix Heide},
  title = {{Inverse Neural Rendering for Explainable Multi-Object Tracking}},
  year = {2024},
  eprint = {2404.12359},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.12359},
}
Figure