A single video camera cannot tell how far away a moving point is. A point drifting rightward on screen could be a nearby object moving slowly or a distant one moving fast, with depth changing in many possible ways along the way. Every method that reconstructs moving 3D scenes from one video, which the field calls monocular 4D reconstruction, has to break this tie somehow. Today they do it by committing early to one answer: a monocular depth estimate, a low-dimensional motion model, or a smoothness penalty. When that answer is wrong, the error is baked into hours of per-scene optimization. We propose to stop committing early. A model reads a 2D point track and its image context, then writes out the identifiers of a handful of real 3D trajectories, drawn from a large library of measured motions, that are all consistent with what the camera saw. The per-scene optimizer then lets the whole video vote among these candidates.
The mechanism comes from generative retrieval: a sequence model is trained to output the discrete ID of a database item token by token, with decoding restricted to IDs that actually exist. This field has mostly been used to find documents and images. Its decoding step, beam search over a prefix tree of valid IDs, already produces a ranked set of distinct candidates. The new ingredient is the ID design. We order each trajectory's ID so that the first tokens describe what a camera can observe, the screen-space path, and the last tokens describe what it cannot, the depth profile given that path. The branching of the beam at those last tokens is then a direct, finite list of the ambiguous alternatives. Each alternative is a motion that really happened somewhere in the library, not a sample from a generator that may produce physically implausible paths.
Core Claim
Depth–motion ambiguity is a one-to-many problem: one 2D track has several plausible 3D lifts (Oh et al.; Attal et al.). Our claim is that a generative retriever with projection-ordered trajectory IDs, decoded under a trie of real trajectory snippets, recovers these alternatives better than embedding-based nearest-neighbor search does. Feeding the alternatives as a multi-hypothesis prior into an existing per-scene optimizer lowers 3D tracking error on truly monocular captures, and it does so most where 2D trackers and depth priors fail. The point is not that retrieval beats optimization. The point is that an explicit, small set of grounded hypotheses gives the optimizer something to choose between instead of something to drift from.
This is distinct from the closest work in the file. DSI-3D maps static LiDAR scans to Hilbert-curve IDs for place recognition. It has no time, no motion, and no ambiguity to resolve. IRGen quantizes ViT class tokens into four-token residual codes for image retrieval and optimizes for a single correct answer. Neither uses the beam as a posterior. On the reconstruction side, Shape of Motion learns ten rigid motion bases from scratch per video and never consults outside motion data. We reuse IRGen's residual-quantization-plus-trie machinery and Shape of Motion's optimizer. What we add is the ID ordering, the multi-hypothesis use of the beam, and the coupling between the two.
Research Program
The trajectory library. We build it from sources with true 3D motion. Synthetic Kubric MOVi-F scenes, the dynamic Objaverse renders used by USplat4D, and human motion capture of the kind behind HumanML3D supply fixed-length snippets of roughly 16 to 32 frames. Each snippet is expressed in a camera frame sampled from realistic handheld paths. Every snippet stores two things: its projected 2D track $(u_t, v_t)$ and its camera-frame depth profile $z_t$, with depth normalized by the snippet's median depth so that global scale is handled separately.
The projection-ordered ID. Each snippet receives an ID of length $L = L_o + L_h$. The first $L_o$ tokens come from residual quantization of the 2D track alone. Residual quantization is a coder that encodes a vector coarsely, then encodes the leftover error, then the leftover of that. The remaining $L_h$ tokens come from residual quantization of the normalized depth profile, conditioned on the 2D code. Trajectories that look identical on screen therefore share a prefix and differ only in their suffix. This applies the locality-preserving intuition of DSI-3D's Hilbert IDs and the hierarchical codes of Ji Won Kim et al. to a different axis: the split between what the camera sees and what it does not.
The retriever. An encoder–decoder transformer reads three things for the query track: the 2D track, a crop sequence of image patches along it, and the monocular depth estimate along it. It also reads the tracks of a few spatial neighbors, so that rigid co-motion can inform depth. The decoder emits ID tokens under trie constraints over the library, following the trie-constrained beam search of IRGen and SEAL. Training uses teacher forcing on the true ID. An added multi-positive objective treats every library snippet whose 2D code matches and whose rendered track lies within tolerance as a valid target, so probability mass is spread across genuinely ambiguous suffixes rather than forced onto one.
The coupling with reconstruction. For each tracked point, the top $K$ beam outputs give candidate depth profiles $\{z^{(k)}\}$ with beam scores $s_k$. Inside a Shape of Motion-style optimizer, the usual depth regularizer is replaced by a mixture penalty:
$$ \mathcal{L}_{\text{hyp}} = -\log \sum_{k=1}^{K} \mathrm{softmax}(s)_k \exp\!\left(-\frac{\lVert \alpha\, z^{(k)} - \hat z \rVert^2}{2\sigma^2}\right). $$
Here $\hat z$ is the optimizer's current depth profile for that point, $\alpha$ is a per-object scale fit jointly, and $\sigma$ is a tolerance. Early in optimization this penalty keeps every point near some real motion. Later, photometric and multi-point rigidity terms pull each point toward the hypothesis that the rest of the video supports. We also log which hypothesis wins per point, which makes the method's choices inspectable.
The analysis. This is half the contribution. We will measure where the beam branches: whether the posterior really becomes multimodal at the depth tokens, and whether that multimodality predicts where single-answer methods fail.
Baselines
The naive reference is per-scene optimization with only off-the-shelf priors. We run Shape of Motion (Wang et al.) exactly as released, with its depth, tracking, and pose inputs. On DyCheck it reports a 3D end-point error of 0.082 against 0.114 for lifting TAPIR tracks with Depth Anything, and the latter is itself a second, even simpler reference. To beat this baseline, our full system must lower 3D end-point error on DyCheck under the same inputs and the same iteration budget. It must also do so on the subset of tracks where TAPIR confidence is low, since that subset is where the claim lives.
The strongest existing reconstruction competitor for rendering is USplat4D (Guo et al.). It propagates motion from reliable key nodes to occluded ones through a spatio-temporal graph and reports 19.63 PSNR on DyCheck at 2x resolution, against 19.32 for MoSca. We do not expect our depth prior to dominate photometric quality. The bar is to match USplat4D's PSNR within noise while improving PCK-T and 3D error, which Gao et al. argue are the metrics that actually reveal collapsed geometry. Because USplat4D's graph and our hypothesis prior are complementary, we also run our prior on top of it.
The strongest competitor for the retrieval component is a dual-encoder retriever. It is trained on the same library with a contrastive loss between the same query encoder and a trajectory encoder, and searched with exact top-$K$ inner product. It receives the same training data, parameter count, and tuning budget, and it plugs into the identical mixture penalty. Our retriever beats it only if it achieves lower best-of-$K$ depth error and higher coverage of ground-truth modes at equal $K$, and if that advantage carries through to lower downstream 3D error.
Three ablations isolate the key ingredients. The first keeps the retriever but uses a standard semantic residual-quantization ID with no observable/hidden split, which tests whether projection ordering matters. The second removes trie constraints, so the decoder freely generates codebook tokens that are decoded to depth profiles. That turns the system into a pure tokenized generative model and tests whether grounding hypotheses in real snippets matters. The third sets $K=1$, which tests whether multiple hypotheses matter at all or whether a better single prior explains any gain.
Why not just…?
Why not just use a dual encoder and FAISS-style nearest-neighbor search over the same library? This is the real threat, and the proposal is built so that it can lose to it. The argument for generative decoding is specific. A dual encoder compresses the query into one vector, and when the true answer is bimodal (near-and-slow versus far-and-fast), a single point in embedding space tends to land between the modes, so its neighbors concentrate around one mode or a blend. The single-vector bottleneck is the limitation Lee et al. identify for late-interaction retrieval, and IRGen reports a 20.2% Precision@10 gain over fine-tuned CLIP in image retrieval. That evidence comes from unimodal targets, however, and says nothing directly about ambiguity. The mode-coverage experiment below settles this. If the dual encoder matches our coverage, we would report that projection-ordered IDs are useful as an analysis tool but that autoregressive decoding adds nothing, and we would recommend the dual encoder.
Why not just train a generative model of trajectories, such as a diffusion model or an unconstrained token generator, and skip the library? A generator also gives multiple hypotheses and avoids the corpus-scaling problem that Hui et al. document for generative retrieval at millions of items. Our bet is that, under weak monocular evidence, a generator's samples drift toward smooth average motions or invent implausible ones. This is the regime where ViDAR reports that diffusion pseudo-supervision cannot repair structural errors such as detached limbs. Trie-constrained retrieval guarantees that each hypothesis is a motion that physically occurred. The unconstrained-decoding ablation tests this directly with matched architecture. If it ties, the library is unnecessary and the cheaper generator wins.
Why not just improve the per-scene motion model, with more bases, splines, or better graphs, and use no outside data? SplineGS, WorldTree, and USplat4D all refine the motion representation within one video. When a part is fully occluded or the tracker loses it, however, no amount of within-video structure supplies the missing depth. Gao et al. show large drops in PSNR and PCK-T under true monocular capture despite such priors. Our stratified evaluation, which splits tracks by tracker confidence and visibility, shows whether outside motion data helps exactly where within-video structure has nothing to work with. If gains appear only on well-tracked points, this alternative is sufficient.
Proposed Experiments
The first experiment checks ambiguity coverage before any reconstruction, and it decides whether the project continues. On held-out MOVi-F and Objaverse dynamic snippets, we construct ground-truth ambiguity sets by searching the library for snippets whose projections match within a pixel tolerance. We then measure, for each method and each $K$ from 1 to 16, the fraction of these modes hit and the best-of-$K$ normalized depth error. The methods are our retriever, the dual encoder, the unconstrained generator, and the non-ordered-ID retriever. We found no reported result on Lacuna for MOVi-F 3D tracking from any cited method, so these numbers will be new. If our retriever does not cover clearly more modes than the dual encoder at equal $K$, we stop and report that negative result.
The second experiment is downstream reconstruction on DyCheck iPhone with co-visibility masking. We report 3D end-point error against Shape of Motion's 0.082, plus PCK-T, PSNR against USplat4D's 19.63, and LPIPS. Each is reported overall and stratified by TAPIR confidence and occlusion duration. None of the library comes from DyCheck, so there is a real synthetic-to-real gap. We will report it honestly rather than fine-tune on the test domain. On NVIDIA-LS and HyperNeRF we report rendering metrics as a check that the prior does not hurt scenes with mild ambiguity, using Chen et al.'s 25.78 mean PSNR on HyperNeRF (Chen et al.) as the reference.
The third experiment covers cost and scale. We measure wall-clock time to reach Shape of Motion's final 3D error, against its reported two hours, since a good initialization may shorten optimization. We grow the library from 100 thousand to several million snippets to check whether the scaling degradation Hui et al. found for text corpora appears here, with the dual encoder as the control.
The decisive check combines these results. The proposal succeeds if three things hold together: higher mode coverage than the dual encoder, lower 3D error than Shape of Motion concentrated on low-confidence tracks, and a measurable loss from removing either the trie constraint or the projection ordering. Any one of these failing tells us which simpler alternative the field should use instead.
References
- Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
- TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis
- DSI-3D: Differentiable Search Index for Point Cloud Retrieval
- IRGen: Generative Modeling for Image Retrieval
- Shape of Motion: 4D Reconstruction from a Single Video
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- Developing Generative Recommender Systems for Government Subsidy Pro-Grams with a New RQ-VAE Model: Wello & the Korean Government Case
- Autoregressive Search Engines: Generating Substrings as Document Identifiers
- Monocular Dynamic View Synthesis: A Reality Check
- Generative Multi-hop Retrieval
- How Does Generative Retrieval Scale to Millions of Passages?
- ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs
- SplineGS: Robust Motion-Adaptive Spline for Real-Time Dynamic 3D Gaussians from Monocular Video
- WorldTree: Towards 4D Dynamic Worlds from Monocular Video using Tree-Chains
- From Tokens to Nodes: Semantic-Guided Motion Control for Dynamic 3D Gaussian Splatting