A single camera cannot tell a small nearby car from a large distant one. Every pixel sits on a ray of possible 3D points, and depth error is the main thing holding back camera-only 3D perception: one headroom study found that substituting ground-truth depth lifts monocular detection mAP from about 20% to over 60%, while substituting true size or orientation barely helps (Depth Estimation Matters Most). The best recent depth fixes work by repeated correction. RefineAny3D draws the predicted 3D box as a wireframe on the image and asks a vision-language model whether to move it "closer" or "farther," by a small, medium or large step, and repeats until the fit looks right. That loop costs about 1.62 sequential model calls per object. With a dozen objects per frame, that rules it out for real-time driving.
We propose to make the tracker's own frame-to-frame recurrence do this refinement. Each tracked object carries a depth estimate and an uncertainty. On every new frame, the network takes exactly one correction step, conditioned on how well the box, moved to account for the camera's own motion, lines up with the object in the image. Many correction steps per frame become one step per frame, and the steps accumulate over the object's lifetime. Each step also sees a new viewpoint, so it adds real parallax evidence instead of re-inspecting the same pixels. The claim is that a single-network monocular tracker built this way can match the accuracy of per-frame iterative refinement after a short warm-up of a few frames, at the cost of one lightweight head per object per frame.
Core Claim and Testable Hypothesis
The idea rests on a specific empirical bet. Per-frame monocular depth errors for a given object are strongly correlated over time: the same car, seen from a similar angle, gets the same scale-driven bias frame after frame. If that holds, smoothing per-frame depths with a Kalman filter or averaging will not remove the bias. A correction that compares the projected box against the object's visible extent can remove it, because a box drawn too small means the depth is too far, whatever happened in earlier frames. That visual cue is what RefineAny3D exploits. Our contribution is to turn its sequential search into a single contractive update inside a recurrent tracking state.
Concretely, let $z_t$ be the log-depth of a track at frame $t$, and let $\tilde z_{t}$ be $z_{t-1}$ moved into frame $t$ using ego-motion. The camera pose comes from odometry, as in PRT-Fusion's ego-motion compensation. A small head reads image features sampled along the projected wireframe edges, together with the track embedding, and outputs an update $\Delta_t$, so that $z_t = \tilde z_t + \Delta_t$. During training we perturb depth along the camera ray, in the spirit of TQD-Track's noise injection but restricted to the ambiguous ray direction. We then require each step to shrink the error by a fixed factor:
$$ \mathcal{L}_{\text{contract}} = \max\big(0,\; |z_t - z^\star_t| - \rho\,|\tilde z_t - z^\star_t|\big) + \lambda\,|z_t - z^\star_t|, $$
Here $z^\star_t$ is the ground-truth log-depth, $\rho<1$ is the target per-frame contraction rate, and $\lambda$ weights a plain regression term. Under this loss, error after $k$ frames should fall roughly as $\rho^k$ until it reaches the floor set by image evidence, since each step is trained to contract by $\rho$. The predicted uncertainty decides how far a step may move and when a track counts as "settled." This plays the role of RefineAny3D's <depth_ok> early-stopping token, but costs nothing extra to compute.
Research Program
The base system is a query-based monocular temporal detector. We use MAGneT-3D's dynamic anchors, which predict 2D centers and focal-normalized "virtual" depths from each frame instead of relying on fixed learned queries. That choice matters because MAGneT-3D showed fixed queries overfit to the training camera. We add persistent track queries in the style of TrackFormer, so identities carry across frames inside one network rather than through a separate association stage. This also fills the gap MAGneT-3D leaves open: it has no explicit track lifecycle. The new piece is the alignment-conditioned depth update attached to each track query.
We will build four variants of the update to separate what matters. The first applies the update once per frame with the track state (the proposal). The second applies the same head $K$ times on the current frame only, which is the per-frame iterative upper reference. The third is a single-shot residual depth head with no track state. The fourth is a recurrent update with the wireframe-alignment features removed. Optionally, we will distill RefineAny3D's discrete actions into the head, in the spirit of how DCFold collapses AlphaFold3's recycling and denoising into one pass and how Meta-Reflection internalizes critique loops. This tests whether a model that reasons about object size teaches anything that ground-truth supervision does not. Both of those papers motivate collapsing a loop into a single pass. Neither moves the loop's steps onto an axis, time, that the system already has to step through.
The main analysis plots per-object depth error against track age, split by distance and by occlusion level. We will also measure, on validation sequences, the autocorrelation over time of per-frame depth residuals from the base detector. That second measurement checks the core premise directly.
Baselines
The naive reference is the decoupled pipeline. The base detector's per-frame 3D boxes are associated with a 3D Kalman filter and Hungarian matching, using identical detector weights. It shows what we gain over not unifying the pipeline at all.
The strongest monocular-depth competitor is RefineAny3D run per frame on the same detector's boxes, with its reported VLM and protocol. It reports Moderate $AP_{3D}$ on KITTI rising from 23.98 to 27.47 on MonoCoP, and $AP_{3D}$ on Omni3D rising from 34.38 to 38.73 on DetAny3D. We will report its accuracy alongside its measured wall-clock time per frame on the same GPU. Beating it means matching its per-object depth error for tracks at least three frames old while running in real time. We do not expect to beat it on newly born tracks, and we will say so.
For temporal monocular detection, MAGneT-3D reports in-domain nuScenes NDS of 32.9 and zero-shot average NDS of 18.6 across Waymo, Lyft and ONCE. Our system shares its anchor generator, so any gain isolates tracking plus refinement.
For tracking, TQD-Track reports nuScenes test AMOTA of 0.511. That result is not cleanly monocular, so we cite it as an upper context rather than a like-for-like target. QD-3DT is the closest monocular 3D multi-object tracker, combining a recurrent VeloLSTM motion model with association in a multi-stage pipeline. No reported AMOTA number for QD-3DT was found on Lacuna, so we will rerun its released protocol.
All learned variants share the backbone, training data, schedule and tuning budget. The four update variants above are the ablations that isolate the key ingredient: alignment-conditioned updates spread over time.
Why not just…?
Why not just smooth per-frame depth with a Kalman filter along each track? This is the most serious alternative, and it is cheap. If per-frame depth errors are roughly independent over time, filtering will remove most of the error and our head adds little. Our premise is that the errors are correlated, because the same object is mislabeled the same way. The residual-autocorrelation measurement and a head-to-head against Kalman smoothing on identical detections settle this directly. If autocorrelation turns out low and Kalman matches us, we will report that the temporal half of the idea is unnecessary.
Why not just train a better single-shot depth residual head? MonoDETR shows that object-level depth supervision fed into queries is strong. A residual head trained on the same ray-perturbed boxes might capture most of RefineAny3D's gain without any recurrence. The third ablation variant is exactly this model. The proposal only stands if error keeps falling with track age beyond what the single-shot head reaches, especially for distant objects, where MAGneT-3D reports the steepest degradation.
Why not just plug in a depth foundation model? RefineAny3D reports that such models give worse object-level box depth than a detector's own heads, because they model scene layout rather than metric object extents. They also add a large network to the latency budget. We will include one as an extra depth input to the naive pipeline to confirm this in our setting.
Proposed Experiments
We will train on nuScenes, following MAGneT-3D's monocular protocol, and on KITTI tracking. We will test zero-shot on Waymo to check that the update generalizes across cameras. Metrics are AMOTA, ID switches and NDS on nuScenes, $AP_{3D}$ at Moderate difficulty on KITTI, absolute relative depth error by distance bin and track age, and frames per second on one GPU.
Four checks are decisive. First, depth residuals from the base detector must show substantial autocorrelation over time. Second, depth error for the proposed model must decay with track age and approach the error of the $K$-step per-frame variant within a few frames. Third, the proposed model must beat Kalman smoothing and the single-shot residual head on far-range and occluded objects. Fourth, its runtime must stay within roughly one decoder pass of the base detector, while per-frame RefineAny3D does not. If the first check fails, we will drop the recurrence and report the single-shot head as the correct and simpler answer.
References
- Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking
- RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection
- TQD-Track: Temporal Query Denoising for 3D Multi-Object Tracking
- MAGneT-3D: Monocular and Domain-Generalizable Temporal 3D Detection
- TrackFormer: Multi-Object Tracking with Transformers
- DCFold: Efficient Protein Structure Generation with Single Forward Pass
- Meta-Reflection: A Feedback-Free Reflection Learning Framework
- Monocular Quasi-Dense 3D Object Tracking
- MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection