TL;DR
TIRE (Track, Inpaint, REsplat) is a novel three-stage framework designed to solve the problem of identity loss in subject-driven 3D and 4D generation. By converting the complex task of 3D texture consistency into a progressive 2D tracking and personalized inpainting problem, TIRE successfully preserves a subject's unique features across all viewpoints. This approach significantly outperforms existing state-of-the-art generative models, which frequently hallucinate incorrect colors or textures on previously unobserved angles.
What Problem Does This Paper Address?
Recent breakthroughs in feed-forward 3D and 4D generation (like L4GM, TRELLIS, and Hunyuan3D) can create photorealistic assets in seconds. However, these models suffer from a major flaw when it comes to personalization: they struggle to maintain the semantic identity of a specific subject across different viewpoints.
When given a single reference image or video of a subject (e.g., a specific tabby cat), feed-forward models have to hallucinate the appearance of unobserved angles (like the cat's back). Due to inherent dataset biases, these models often introduce systematic errors—such as turning the back of a normal cat completely blue or over-smoothing fine details.
While optimization-based methods like Score Distillation Sampling (SDS) exist, they are painfully slow and tend to produce over-smoothed, "averaged-out" results. TIRE aims to bridge this gap by acting as a model-agnostic plug-in that refines rough 3D/4D assets to preserve identity accurately.
Methods
TIRE takes a rough 3D asset generated by an existing model and applies a three-stage pipeline: Track, Inpaint, and Resplat.
- Stage I: Track (Mask Identification): The framework renders a 360-degree video sequence of the rough asset. It then uses a video tracking model (CoTracker) to establish pixel correspondences between the target views and the reference source view. Instead of tracking forward, it uses a clever backward tracking mechanism to identify exactly which regions in the novel views are unobserved and need infilling.
- Stage II: Inpaint (Progressive Personalization): TIRE injects LoRA weights into a pretrained Stable Diffusion inpainting model and fine-tunes it on the reference view to make it subject-driven. To handle views far from the source, it infills textures progressively. It first inpaints a "sweet spot" ($\pm20^\circ$), uses those newly generated high-quality regions as anchor points to track and inpaint further angles ($\pm90^\circ$), and repeats until the entire asset ($\pm180^\circ$) is covered.
- Stage III: Resplat (3D Lifting & Consistency): Because 2D images inpainted frame-by-frame can suffer from multi-view inconsistency, TIRE passes the images through a multi-view diffusion model (like Wonder3D) using a mask-aware latent update strategy. Finally, these consistent 2D views are unprojected ("resplatted") back into 3D Gaussians.
Datasets
To evaluate subject-driven 3D/4D generation, the authors introduced the DreamBooth-Dynamic benchmark dataset.
- Composition: It features 23 animatable subjects curated from the original 2D DreamBooth dataset.
- Creation: The authors trained a customized text-to-image model (SDXL + LoRA) for each subject, generated static scenes with specific prompts, animated them using the CogVideoX video generation model, and segmented the foregrounds using SAM 2.
- Evaluation: The framework was also tested on complex, in-the-wild data from the official L4GM project website.
Key Insights and Ideas
- The Power of Backward Tracking: Intuitively, one might track pixels forward from the source view to target views. However, the authors discovered this creates grainy, fragmented masks. Backward tracking (from target views back to the source) leverages the fact that the source view contains the richest identity information, yielding cleaner, well-shaped masks that drastically improve inpainting quality.
- The "Sweet Spot" Anchor: Jumping directly from a front-facing view to a back-facing view is too difficult for 2D inpainting models. By establishing a progressive pipeline using a $\pm20^\circ$ "sweet spot" as a stepping stone, the model balances exploration of unseen data with the exploitation of reliable, near-source details.
- Flaws in Standard Metrics: The paper highlights that standard vision metrics like DINO feature similarity can fail when evaluating 3D geometry. A baseline model can score highly on DINO if its 2D texture matches the reference, even if its 3D geometry is completely broken (e.g., putting a flat face on the back of a head). This underscores the necessity of using Vision-Language Models (VLMs) and human user studies for 3D evaluation.
Key Results
- VLM-Based Evaluation: Evaluated across six different Vision-Language Models (including GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL), TIRE achieved the highest average subject consistency score (1.854 out of 4) compared to top-tier baselines like Hunyuan3D-v2.5 (1.703) and TRELLIS (1.476).
- User Study: In a blind user study with 18 participants scoring overall asset quality from 1 to 10, TIRE scored 6.37, comfortably beating L4GM (5.67) and Customize-It-3D (1.72).
- Geometry Refinement: Beyond texture, TIRE implicitly improves the underlying geometry of the assets, significantly reducing the "ghosting" artifacts common in multi-view inconsistent models.
Limitations and Future Work
- Computational Efficiency: TIRE is not yet a real-time feed-forward solution. Due to the test-time fine-tuning required by the LoRA-based inpainting stage, it takes approximately 100 minutes on a single NVIDIA A100 GPU to optimize an asset. The authors anticipate that future feed-forward 2D subject-driven inpainting models will drastically speed up this pipeline.
- Temporal Dimension Constraints: Currently, TIRE treats identity as a static attribute. For 4D (video) generation, it does not explicitly model complex, changing temporal patterns over time (e.g., a pattern that shifts color as an animal moves). Exploring time-varying identity preservation remains an open avenue for future research.
Summary updated