TL;DR
TIRE (Track, Inpaint, REsplat) is a novel framework that brings high-fidelity personalization to 3D and 4D generation. By progressively "fixing" the unobserved parts of a generated object using video tracking and subject-driven 2D inpainting, TIRE ensures that a generated asset—like a specific pet—maintains its unique identity and textures from every possible angle.
The Challenge of Personalization in 3D/4D Generation
Current state-of-the-art 3D and 4D (animated 3D) generation models are incredibly fast and produce photorealistic results. However, they struggle significantly with personalization.
When provided with a single image of a specific subject (e.g., your dog wearing a unique harness), these models can easily generate an accurate front view. But as the camera rotates to the side or back, the model often loses the subject's identity. It tends to revert to generic features or produce "ghosting" artifacts and unrealistic colors—such as a strange blueish tint on occluded regions. This occurs because the model lacks sufficient cues to infer the appearance of unobserved viewpoints, defaulting instead to biased patterns learned during training.
The TIRE Pipeline: Track, Inpaint, Resplat
To solve this, the authors introduce TIRE, a three-stage pipeline designed to "infill" the missing identity of a rough 3D asset:
- Track: The system takes an initial rough 3D asset and renders a 360° video. It uses a video tracking model to identify which pixels in novel views correspond to the original source image. Crucially, TIRE employs backward tracking—tracking from target views back to the source—which produces much cleaner, more accurate masks for the regions that need modification.
- Inpaint: To fill in the "unknown" regions identified by the masks, the authors personalize a 2D diffusion inpainting model using LoRA (Low-Rank Adaptation) on the user's source image. They use a progressive strategy: first inpainting views close to the source (the "sweet spot"), then using those newly generated views as anchors to inpaint further viewpoints (like the direct back of the object).
- Resplat: Finally, the modified 2D multi-view observations are projected back into 3D space. The system uses Large Gaussian Models to "resplat" these pixels into a consistent 3D Gaussian representation, ensuring the final asset is spatially and temporally coherent.
Datasets and Evaluation
The researchers evaluated TIRE on diverse datasets, including a newly introduced benchmark:
- DreamBooth-Dynamic: A new benchmark created by taking subjects from the original DreamBooth dataset, animating them with CogVideoX, and segmenting them with SAM 2 to create high-quality source data for subject-driven 4D generation.
- In-the-wild data: The model was also tested on diverse, uncurated examples from existing 4D generation projects to demonstrate real-world robustness.
Key Insights
- Backward Tracking is Superior: While forward tracking (source to target) is more intuitive, it often results in "grainy" masks. Backward tracking leverages the fact that the source view contains the richest identity information, establishing better correspondences and cleaner boundaries for inpainting.
- The "Sweet Spot" Anchor: Jumping directly from a front view to a back view in one step usually results in a loss of quality. By inpainting ±20° views first, the model creates "anchor" viewpoints that bridge the gap to the more challenging 180° views.
- Model Agnostic: TIRE acts as a "plug-in" refinement stage. Whether the initial 3D model was made using NeRFs, Meshes, or Gaussians, TIRE can process the rendered frames and lift them back into a high-fidelity 3D representation.
Results
TIRE significantly outperforms existing baselines like L4GM, SV3D, and TRELLIS in preserving subject identity.
- User Study: In a blind test, human volunteers rated TIRE's overall quality at 6.37/10, compared to 5.67 for L4GM and a mere 1.72 for Customize-It-3D.
- VLM Evaluation: Using Vision-Language Models (like GPT-4o and Gemini 2.0) to score subject consistency, TIRE achieved an average score of 1.854/4, outperforming Hunyuan3D-v2.5 (1.703) and TRELLIS (1.476).
- Qualitative Improvements: TIRE successfully removes the "hallucinated" tones and generic textures found in baseline models, replacing them with textures that faithfully match the source subject.
Limitations and Future Work
While highly effective, TIRE has a few limitations. The current pipeline takes approximately 100 minutes on a single NVIDIA A100 GPU. While faster than some optimization-based methods, it is slower than pure feed-forward models. Additionally, TIRE relies on a "rough" initial 3D asset; if the base model produces completely broken geometry, the 2D-to-3D lifting may struggle to fix it.
Future work will likely move toward a fully feed-forward personalization model to eliminate the need for per-subject finetuning, further reducing generation time.
Summary updated