TL;DR
SceneAnything is an automated framework that transforms a single, casual video into a fully editable 3D scene composed of separate mesh objects. By coordinating a "team" of AI agents—including a scene owner, authors, and judges—it bridges the gap between raw geometric estimates and structured, semantically meaningful 3D environments that can be relit, modified, and explored from any angle.
What Problem Does This Paper Address?
Reconstructing 3D scenes from video has traditionally fallen into two camps:
- Geometric methods (like 3D Gaussian Splatting or NeRF): These produce stunningly realistic renders but result in "frozen" volumes. You can't easily move a chair, change the wallpaper, or adjust the sun's position because the scene isn't made of distinct objects.
- Agentic methods: Recent attempts have used Large Language Models (LLMs) to write Blender scripts to "reconstruct" a scene. However, without a strict grounding in reality, these agents often hallucinate layouts, get scales wrong, or fail to match the lighting of the original video.
The challenge is creating a system that is both geometrically accurate (matching the video) and semantically editable (composed of separate, named mesh objects with realistic materials).
Methods
The core of SceneAnything is a software harness that orchestrates a multi-agent workflow, grounding high-level reasoning in low-level vision data.
The process follows a structured protocol:
- Geometric Grounding: The system uses pre-trained models like Lyra 2 for camera and depth estimation and SAM 3.1/Mask2Former for segmentation. This provides the "ground truth" measurements that agents must follow.
- The Scene Owner Agent: This agent plans the environment, identifies objects, and writes "briefs" for other agents.
- The Author Agents: These agents receive briefs containing measurements and coordinates. They write parameterized Blender programs to generate geometry and materials.
- Level-of-Detail (LoD) Policy: To avoid over-engineering, the system uses a visibility-based rule. If an object is far away or blurry, it uses a simple procedural recipe; if it's close and clear, it's built part-by-part.
- The Judge Agents: To ensure quality, separate agents act as "blind" judges. They compare renders of the reconstructed scene against the original video frames. A revision is only accepted if the judges agree it's an improvement.
Datasets
The authors evaluated SceneAnything on a diverse set of 100 scenes, including:
- Real-world videos: 39 static and 20 moving-element videos from Wikimedia Commons.
- Generated videos: 17 videos created by AI video generators.
- Standard benchmarks: 24 photo sets from established datasets like Mip-NeRF 360 and LLFF.
The system was compared against direct-agent baselines (GPT-4, Claude) and state-of-the-art video-to-3D methods like Pi3-X and 3D Gaussian Splatting (3DGS).
Key Insights and Ideas
- Agents Need a Manager: Simply asking an LLM to "make this scene" fails. The "harness" is essential because it provides the agents with a shared coordinate system and physical constraints (like gravity and scale).
- Blind Judging is Crucial: The authors found that agents are often too "proud" of their own work. Using independent judge agents who don't know which render is the "new" one prevents the system from accepting low-quality hallucinations.
- Procedural Completion: Unlike point-cloud methods that leave holes where the camera didn't see, SceneAnything's agents use architectural logic to "fill in" the backs of buildings or hidden rooms, making the scene truly explorable.
Key Results
SceneAnything significantly outperforms both "naive" agentic reconstruction and traditional video-to-3D methods in terms of editability and novel-view consistency.
- Fidelity: In original-view comparisons, SceneAnything achieved an LPIPS score of 0.50, outperforming the best baseline (0.56).
- Novel Views: When moving the camera far from the original path, SceneAnything maintained high content similarity (DINOv3 score of 0.53), whereas methods like 3DGS often "break" or smear (scoring 0.38 or lower).
- Editability: Because the output is a Blender file, users can perform complex edits like relighting a sunset or removing objects, which is impossible with Gaussian Splatting.
Limitations and Future Work
- Model Dependency: The system's quality is heavily tied to the underlying LLM. While it works well with frontier models like Claude 5.5 Opus, performance drops with smaller open-source models.
- Human Reconstruction: The current system does not attempt to reconstruct humans, which remains a notoriously difficult problem in 3D vision.
- Non-Rigid Motion: While it handles water surfaces well, complex non-rigid motion (like a waving flag or a person walking) is currently outside the system's scope.
- Single Shots: The system currently processes one continuous video shot at a time; unifying multiple shots with cuts into a single scene is a goal for future work.
Summary updated