SceneAnything: Agentic Reconstruction of Large-Scale Mesh Scenes from a Single Video

Bowen Xue, Saeed Hadadan, Fabrice Rousselle, Zahra Montazeri, Ravi Ramamoorthi, Milos HasanarXiv 2026 · arXiv

TL;DR

SceneAnything is an automated framework that transforms a single, casual video into a fully editable 3D scene composed of separate mesh objects. By coordinating a "team" of AI agents—including a scene owner, authors, and judges—it bridges the gap between raw geometric estimates and structured, semantically meaningful 3D environments that can be relit, modified, and explored from any angle.

What Problem Does This Paper Address?

Reconstructing 3D scenes from video has traditionally fallen into two camps:

  1. Geometric methods (like 3D Gaussian Splatting or NeRF): These produce stunningly realistic renders but result in "frozen" volumes. You can't easily move a chair, change the wallpaper, or adjust the sun's position because the scene isn't made of distinct objects.
  2. Agentic methods: Recent attempts have used Large Language Models (LLMs) to write Blender scripts to "reconstruct" a scene. However, without a strict grounding in reality, these agents often hallucinate layouts, get scales wrong, or fail to match the lighting of the original video.

The challenge is creating a system that is both geometrically accurate (matching the video) and semantically editable (composed of separate, named mesh objects with realistic materials).

Methods

The core of SceneAnything is a software harness that orchestrates a multi-agent workflow, grounding high-level reasoning in low-level vision data.

Figure 2. The stages of SCENEANYTHING; the pictures are artefacts of the Winter Palace run. Above the band: each stage’s models and tools. The band: the harness and its rules. Below: what each agent does and, in the darker cell, what it decides for the scene.

The process follows a structured protocol:

  • Geometric Grounding: The system uses pre-trained models like Lyra 2 for camera and depth estimation and SAM 3.1/Mask2Former for segmentation. This provides the "ground truth" measurements that agents must follow.
  • The Scene Owner Agent: This agent plans the environment, identifies objects, and writes "briefs" for other agents.
  • The Author Agents: These agents receive briefs containing measurements and coordinates. They write parameterized Blender programs to generate geometry and materials.
  • Level-of-Detail (LoD) Policy: To avoid over-engineering, the system uses a visibility-based rule. If an object is far away or blurry, it uses a simple procedural recipe; if it's close and clear, it's built part-by-part.
  • The Judge Agents: To ensure quality, separate agents act as "blind" judges. They compare renders of the reconstructed scene against the original video frames. A revision is only accepted if the judges agree it's an improvement.

Datasets

The authors evaluated SceneAnything on a diverse set of 100 scenes, including:

  • Real-world videos: 39 static and 20 moving-element videos from Wikimedia Commons.
  • Generated videos: 17 videos created by AI video generators.
  • Standard benchmarks: 24 photo sets from established datasets like Mip-NeRF 360 and LLFF.

The system was compared against direct-agent baselines (GPT-4, Claude) and state-of-the-art video-to-3D methods like Pi3-X and 3D Gaussian Splatting (3DGS).

Key Insights and Ideas

  • Agents Need a Manager: Simply asking an LLM to "make this scene" fails. The "harness" is essential because it provides the agents with a shared coordinate system and physical constraints (like gravity and scale).
  • Blind Judging is Crucial: The authors found that agents are often too "proud" of their own work. Using independent judge agents who don't know which render is the "new" one prevents the system from accepting low-quality hallucinations.
  • Procedural Completion: Unlike point-cloud methods that leave holes where the camera didn't see, SceneAnything's agents use architectural logic to "fill in" the backs of buildings or hidden rooms, making the scene truly explorable.
Figure 3. The level-of-detail rule (Eq. 4) at two scales: two garden beds and a city filmed from a hill (last row). Without the rule, every plant of a bed is one instance of its family’s generator: it has leaves of the right kind but not the leaves of the frame. In the city frame, the elements built part by part cover 77% of the building pixels.

Key Results

SceneAnything significantly outperforms both "naive" agentic reconstruction and traditional video-to-3D methods in terms of editability and novel-view consistency.

  • Fidelity: In original-view comparisons, SceneAnything achieved an LPIPS score of 0.50, outperforming the best baseline (0.56).
  • Novel Views: When moving the camera far from the original path, SceneAnything maintained high content similarity (DINOv3 score of 0.53), whereas methods like 3DGS often "break" or smear (scoring 0.38 or lower).
  • Editability: Because the output is a Blender file, users can perform complex edits like relighting a sunset or removing objects, which is impossible with Gaussian Splatting.
Figure 1. SCENEANYTHING turns a casually recorded or generated video into a 3D scene made of separate, editable mesh objects. Each row is one video. The agents of columns 3 and 4 are each prompted directly to reconstruct the video using Blender; 3DGS [18] is trained on the video without the frame of column 1. Working alone, the agents produce plausible objects, but with simplified shapes and suboptimal layout, scale and lighting. Our method achieves a much better fit in terms of geometry, materials and lighting. As expected, 3DGS fits the video well (column 5) but fails at a new camera far from the filmed path (column 7).
Demonstration of relighting and object manipulation (moving a catamaran) within the reconstructed scene.

Limitations and Future Work

  • Model Dependency: The system's quality is heavily tied to the underlying LLM. While it works well with frontier models like Claude 5.5 Opus, performance drops with smaller open-source models.
  • Human Reconstruction: The current system does not attempt to reconstruct humans, which remains a notoriously difficult problem in 3D vision.
  • Non-Rigid Motion: While it handles water surfaces well, complex non-rigid motion (like a waving flag or a person walking) is currently outside the system's scope.
  • Single Shots: The system currently processes one continuous video shot at a time; unifying multiple shots with cuts into a single scene is a goal for future work.

Summary updated

Abstract

Reconstructing an editable 3D scene from video requires matching visible structure while plausibly completing unseen surfaces. Agents can write scene-construction programs, but direct reconstruction attempts often produce errors in layout, scale, and lighting. We introduce SceneAnything, an automated system for reconstructing mesh scenes from real and generated videos, enabling text-to-scene construction when paired with video generation. A software harness coordinates agents, pretrained vision models, and graphics tools through an explicit reconstruction protocol. Shared camera, depth, and segmentation estimates ground component construction: a scene-owner agent plans the environment and briefs authors, who write parameterized Blender programs. A visibility- and fit-based level-of-detail rule selects the construction level, while unseen regions are completed and marked as inferred. Assembled scenes use fitted lighting and supported motion, and revisions undergo programmatic checks and independent visual comparisons. Accepted programs and lessons are reused across scenes. We evaluate on 100 scenes; subset comparisons show improved original-view fidelity over direct-agent reconstruction and higher novel-view content similarity than the compared video-to-3D methods. We will release the harness, agent instructions, reusable libraries, and scenes.
BibTeX
@misc{xue2026sceneanything,
  author = {Bowen Xue and Saeed Hadadan and Fabrice Rousselle and Zahra Montazeri and Ravi Ramamoorthi and Milos Hasan},
  title = {{SceneAnything: Agentic Reconstruction of Large-Scale Mesh Scenes from a Single Video}},
  year = {2026},
  eprint = {2611.25340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25340},
}
Figure