How Much Picture Does a Markerless Tracker Need?

Augmented reality overlays and robot grasping both depend on knowing an object's 3D position and orientation, its six-degree-of-freedom (6DoF) pose, in every video frame. The traditional way to get it is to stick a printed fiducial marker, a high-contrast square tag, on the object. Markers are mature, but they break as soon as they are dirty, occluded, or impossible to attach. Markerless trackers avoid them with "render-and-compare": the tracker draws a picture of the object at its last estimated pose, then nudges the pose until the picture lines up with the camera frame. The field has invested heavily in making that picture more realistic. PixTrack, for example, replaces CAD models with neural radiance fields, 3D scene models learned from photographs that can be rendered from any viewpoint. Recent image generators build pictures coarse-to-fine, one resolution level at a time. That suggests a cheaper option: render only the coarse levels and add fine detail only when it helps.

Nobody has measured which parts of the rendered picture actually move pose accuracy, and the standard metric could not show it if they tried. The usual score, average distance of model points (ADD), is dominated by depth error. It hides the rotation errors that make an overlay look visibly wrong. This proposal builds a protocol-locked evaluation harness for markerless tracking and uses it to run a scale-resolved template study. It tests three things: whether realism at coarse and fine scales has different effects on translation and rotation accuracy, whether a coarse-to-fine learned generator buys anything over simply blurring a normal render, and whether any learned template beats a well-tuned classical geometric tracker at all.

Core Claim

We hypothesize that, in render-and-compare tracking, coarse template content (silhouette and large-scale shading) accounts for nearly all of the achievable ADD accuracy. Fine-scale appearance fidelity would matter only for reprojection error, the pixel misalignment of the projected model. If so, a tracker can spend high-resolution synthesis only in its last few iterations at little cost in accuracy. We also expect this benefit to appear on textured objects and to disappear on textureless ones.

The hypothesis can fail in three informative ways. Fine detail may not help reprojection error either. A plain blurred render may match a learned hierarchical generator at every budget. Or classical contour-based trackers may dominate everything, which would mean the realism race is not buying accuracy. The deliverable is a measurement, and each of these outcomes answers a question the field currently answers by assertion.

Grounding in Prior Work

Two papers motivate the evaluation side. For A More Comprehensive Evaluation of 6DoF Object Pose Tracking showed three problems with how tracking is scored. The original YCB-Video annotations err by 21.4mm on average; the authors re-annotated them down to 0.85mm. Papers re-initialize trackers inconsistently after failure. And ADD falls into a "translation trap," so reprojection error (PRJ) has to be reported alongside it. On refined YCB-Video, the hybrid of the classical trackers ICG and SRT3D reached the best ADD-PRJ-AUC. BCOT supplies markerless ground truth under 2mm for 20 textureless objects across 404 sequences. It found that trackers rendering templates online beat trackers that rely on pre-rendered views. We reuse both datasets and the non-reinitialized protocol. Neither paper asks why rendering helps, or which rendered content matters.

NerfBaselines supplies the engineering method. It wraps official codebases in containers and fixes color range, downscaling and metric versions. It exposed that 12 of 19 view-synthesis codebases no longer install from their official instructions, and that a background-color change can flip rankings. We carry that discipline over to tracking, where it does not yet exist.

On the template side, PixTrack is the direct method ancestor. It renders a radiance-field template, extracts learned features, and aligns them by Levenberg-Marquardt optimization. It reports ADD-S of 0.95 (RGB) and 0.98 (RGB-D), but on the original, mis-annotated YCB-Video and without reprojection error. Its central claim, that realistic templates close the CAD domain gap, is therefore untested at the precision where the domain gap should matter. Rambach et al. render rasterized textured scans and deliberately remove appearance with a pencil edge filter. That is implicit evidence that coarse structure may suffice; they evaluated only on a custom industrial testbed.

The hierarchical generators supply the scale-controlled instrument. VAR predicts whole token maps at increasing resolution. MS-VQVAE sums per-scale residual decoders, which permits a coarse "preview" exit. We borrow the MS-VQVAE residual structure because it lets us truncate a template at scale $k$ by construction. ZebraPose shows that coarse-to-fine discrete surface codes help single-frame pose estimation. We ask the analogous question for temporal tracking, but through the appearance of the rendered template rather than through the prediction target.

Research Program

The first component is the harness. It containerizes official implementations of ICG, SRT3D, their hybrid, LDT, PixTrack, and a rasterized-CAD render-and-compare tracker. It evaluates them all on refined YCB-Video and BCOT under one protocol: non-reinitialized subsequences of 25 to 200 frames as in the comprehensive-evaluation paper, a fixed image preprocessing path, and fixed intrinsics handling. It reports ADD-AUC, PRJ-AUC, and rotation and translation error separately. Appearance models are built only from sequences disjoint from the evaluation sequences, with a fixed split published up front, so appearance does not leak from test frames.

The second component is a scale-controlled template family built on a shared alignment loop: PixTrack's feature extractor and optimizer, frozen, with only the template source varied. Three template sources sit in that loop. The first is a rasterized CAD render. The second is a radiance-field render with Laplacian-pyramid truncation at level $k$: a Laplacian pyramid splits an image into a coarse base plus successively finer detail bands, and truncation discards the bands above $k$. The third is a learned hierarchical template. For it, a multi-decoder residual generator in the MS-VQVAE style takes the CAD geometry buffers (depth, normals, silhouette) at the current pose and outputs appearance scale by scale. It is trained per object on the disjoint appearance sequences and truncated at scale $k$.

We then sweep $k$ jointly with an iteration schedule that decides at which alignment step fine scales are switched on. Each tracker produces an accuracy-versus-template-compute curve, separately for rotation and translation and separately for textured (YCB-Video) and textureless (BCOT) objects.

Baselines

The naive reference point is a tracker that never updates its template: a single pre-rendered keyframe at the initial pose. It bounds how much online rendering matters at all. BCOT's finding that pre-rendered views fail under rotation predicts this baseline will be weak, and the harness will confirm or overturn that under the locked protocol.

The strongest existing method is the classical hybrid ICG+SRT3D. It matches object contours and depth to the model with no learned appearance and needs no training data. The comprehensive-evaluation paper reports ADD-PRJ-AUC of 70.3 for it on refined YCB-Video. For BCOT, the file reports only that "LDT algorithm proved most effective," so no reported numeric result was found on Lacuna for LDT or any tracker on BCOT, and we will establish that number in the harness. To keep the comparison fair, every method gets the same subsequences, initial poses, and intrinsics. Classical parameters are tuned on the appearance-split sequences with the same trial budget given to learned templates' hyperparameters, following the lesson of BARS that untuned baselines manufacture progress. A template method beats this baseline only if it exceeds the hybrid's harness-reproduced ADD-PRJ-AUC with a paired bootstrap interval over sequences that excludes zero. Matching the reproduced score is not enough.

PixTrack at full resolution is the strongest learned competitor and the realism ceiling. We rerun it on the refined annotations with PRJ reporting, so its original ADD-S figures are not compared directly.

The ablation that isolates the proposal's key ingredient compares the learned hierarchical template against the Laplacian-truncated radiance-field render at equal truncation level and equal wall-clock budget. If they are indistinguishable, the coarse-to-fine generative structure contributes nothing beyond frequency content. A second ablation, the rasterized CAD render with the pencil edge filter of Rambach et al., tests whether appearance helps at all over structure alone.

Why not just…?

Why not just use fiducial markers? They remain the right answer wherever they can be attached. The markerless regime this proposal targets includes mud-covered machinery (Chikushi et al.), inaccessible scenes (Sandor et al.) and settings where tags are visually intrusive (Uchiyama et al.). Both benchmarks we use are markerless by design, so markers are not a competitor on this question.

Why not just use ICG+SRT3D and skip learned templates? This is the most serious threat. The only precise number in the file says a classical tracker leads, and BCOT's textureless objects offer little appearance to exploit. We do not argue around it. It is the headline baseline, and if it wins on both metrics across both datasets, the conclusion that realistic rendering is not buying tracking accuracy is itself the main result.

Why not just render the radiance field at lower resolution or blur it, with no generative hierarchy? This is a real threat to the hierarchical-synthesis arm. Image pyramids are a standard trick in iterative alignment, and a blurred render already provides a "coarse exit." We therefore make Laplacian truncation the control at matched compute. If it matches the learned hierarchy, we report that the generative structure adds nothing, and the scale-resolved accuracy findings still stand, obtained with the cheaper instrument.

Why not just keep the original YCB-Video with ADD leaderboards? With 21.4mm annotation error, the millimeter-scale rotation effects this study targets fall below the noise floor. ADD also cannot see rotation effects by construction, per the translation-trap analysis. The harness reports both metrics precisely so that this question can be answered.

Decisive Checks

The first check is the rotation–translation split. On refined YCB-Video, truncating templates to the coarsest scales should cost less than one ADD-AUC point relative to full resolution but a clearly larger PRJ-AUC drop. If the two drops are comparable, the core claim fails. The second check is the texture contrast: on BCOT the PRJ gap between coarse and full templates should be substantially smaller than on YCB-Video. A comparable gap would mean the fine scales encode geometry rather than texture. The third check is the schedule: a template that enables fine scales only in the final alignment iterations should match full-resolution PRJ-AUC at a fraction of template compute. The fourth check is against the hybrid ICG+SRT3D score under identical protocol, which decides whether any rendering-based tracker earns its cost. Every container, split and metric script will be released so that each of these checks can be rerun by others, which none of the cited tracking results currently allow.