Prior-free 3D Object Tracking

Xiuqiang Song, Li Jin, Zhengxian Zhang, Jiachen Li, Fan Zhong, Guofeng Zhang, Xueying QinCVPR 2025 · OpenReview

Abstract

In this paper, we introduce a novel, truly prior-free 3D object tracking method that operates without given any model or training priors. Unlike existing methods that typically require pre-defined 3D models or specific training datasets as priors, which limit their applicability, our method is free from these constraints. Our method consists of a geometry generation module and a pose optimization module. Its core idea is to enable these two modules to automatically and iteratively enhance each other, thereby gradually building all the necessary information for the tracking task. We thus call the method as Bidirectional Iterative Tracking (BIT). The geometry generation module starts without priors and gradually generates high-precision mesh models for tracking, while the pose optimization module generates additional data during object tracking to further refine the generated models. Moreover, the generated 3D models can be stored and easily reused, allowing for seamless integration into various other tracking systems. Experimental results demonstrate that BIT outperforms many existing methods, even those that extensively utilize prior knowledge, while BIT does not rely on such information.

Introduction

In computer vision, 3D object tracking involves estimating an object's 6-degree-of-freedom (6DoF) pose—its position and orientation—in real-time. Traditionally, this task has relied on two types of prior knowledge: pre-existing CAD models (model-based tracking) or massive, object-specific training datasets (learning-based tracking). These requirements severely limit the deployment of tracking systems in dynamic, "in-the-wild" environments where novel objects are frequently encountered.

While some "prior-free" methods exist, they often require expensive depth sensors (RGB-D) or sacrifice accuracy. The Bidirectional Iterative Tracking (BIT) framework addresses this by bootstrapping 3D tracking from a standard monocular RGB video stream without any pre-existing knowledge of the object's shape or appearance.

The Bidirectional Iterative Tracking (BIT) Framework

BIT solves the "chicken-and-egg" problem of prior-free tracking: you need a 3D model to track an object, but you need accurate tracking (poses) to build a 3D model. The framework breaks this cycle using two interdependent modules that operate in a continuous loop:

  1. Geometry Generation Module: This module begins with a generic 3D sphere. It utilizes the Segment Anything Model (SAM) to extract the object's silhouette from the background. A differentiable renderer (SoftRas) then deforms the sphere until its 2D projection matches the segmented masks from the video frames.
  2. Pose Optimization Module: This module uses the current 3D mesh to estimate the object's movement in subsequent frames using a region-based tracker.

As the object moves, the tracker identifies "key frames" representing new viewpoints. These viewpoints are fed back into the Geometry module to further refine the 3D mesh. This bidirectional flow ensures that as the mesh becomes more accurate, the tracking becomes more robust, and vice versa.

The BIT framework iteratively refines geometry and pose across multiple video frames to bootstrap tracking from a single RGB source.

Experimental Results

The authors evaluated BIT on the MOPED and RBOT benchmarks, as well as real-world captures. On the MOPED dataset, BIT—despite having no prior knowledge—outperformed several state-of-the-art methods that were specifically trained on the target objects.

On the RBOT benchmark, the tracking accuracy achieved using BIT’s generated models was remarkably close to the performance of trackers using ground-truth CAD models. Furthermore, once the initial model is generated (typically within 30 seconds of video), the system maintains a real-time performance of 30 FPS on standard consumer hardware (NVIDIA GTX 3080).

BIT achieves tracking accuracy comparable to ground-truth CAD models on the RBOT dataset and produces reusable 3D assets.

Key Insights and Future Work

A significant advantage of BIT is the production of reusable assets. Unlike neural-network-based trackers where knowledge is implicit in weights, BIT outputs a standard 3D mesh that can be exported and used by other traditional model-based tracking systems.

However, the system currently faces limitations regarding topology; because it deforms a sphere, it cannot easily represent objects with holes (e.g., a donut). Additionally, the system assumes object rigidity. Future research will focus on extending BIT to handle non-rigid, deformable objects and complex topological structures, further expanding the horizons of truly prior-free computer vision.

Summary updated

Abstract

In this paper, we introduce a novel, truly prior-free 3D object tracking method that operates without given any model or training priors. Unlike existing methods that typically require pre-defined 3D models or specific training datasets as priors, which limit their applicability, our method is free from these constraints. Our method consists of a geometry generation module and a pose optimization module. Its core idea is to enable these two modules to automatically and iteratively enhance each other, thereby gradually building all the necessary information for the tracking task. We thus call the method as Bidirectional Iterative Tracking(BIT). The geometry generation module starts without priors and gradually generates high-precision mesh models for tracking, while the pose optimization module generates additional data during object tracking to further refine the generated models. Moreover, the generated 3D models can be stored and easily reused, allowing for seamless integration into various other tracking systems, not just our methods. Experimental results demonstrate that BIT outperforms many existing methods, even those that extensively utilize prior knowledge, while BIT does not rely on such information. Additionally, the generated 3D models deliver results comparable to actual 3D models, highlighting their superior and innovative qualities. The code is available at https://github.com/songxiuqiang/BIT.git.
BibTeX
@inproceedings{song2025prior,
  author = {Xiuqiang Song and Li Jin and Zhengxian Zhang and Jiachen Li and Fan Zhong and Guofeng Zhang and Xueying Qin},
  title = {{Prior-free 3D Object Tracking}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=maD7IT0rGW},
}
Figure