ManiSplat: Manipulation Trajectory Synthesis from Monocular Video via Decoupled 3D Gaussian Splatting

Wenhao Hu, Haonan Zhou, Liu Liu, Yun Du, Xinjie Wang, Ziang Li, Zhizhong Su, Gaoang WangarXiv 2026 · arXiv

TL;DR

ManiSplat introduces a novel framework to create decoupled, controllable 3D Gaussian digital twins directly from monocular ego-view robotic videos. By organizing the scene into a structured graph and leveraging the inherent phase logic of manipulation tasks, it achieves high-fidelity dynamic reconstruction and enables physically consistent trajectory augmentation for robust robotic policy learning.

What Problem Does This Paper Address?

Bridging the gap between the real world and simulation (Real2Sim) is a major bottleneck in training robotic policies. While 3D Gaussian Splatting (3DGS) has revolutionized static scene reconstruction, extending it to dynamic, interactive environments remains incredibly challenging.

Existing dynamic 3DGS methods typically treat the entire scene as a continuous function of time, meaning they lack object-level understanding and cannot separate a moving object from its background. Furthermore, they struggle with interaction-driven dynamics—such as a robotic arm suddenly grasping a tool. These abrupt pose changes and complex contact interactions often result in severe motion blur, floating artifacts, and geometric instability. ManiSplat tackles these issues by explicitly decoupling the robot, the objects, and the background from a single monocular video demonstration.

Figure 1: High-Fidelity Real2Sim Alignment and Trajectory Augmentation. (Left) Our framework decomposes monocular ego- view video into aligned assets (URDF robot, objects, and back- ground). (Right) This enables a trajectory augmentation pipeline that synthesizes diverse variations from single demonstrations to scale up data for policy learning.

Methods

ManiSplat introduces three core components to achieve high-fidelity, controllable dynamic reconstruction:

  1. Graph-Structured Disentangled Representation: Instead of optimizing a single cloud of Gaussians, the scene is structured as a dynamic scene graph. It separates independent semantic nodes: a static background node, a robot node explicitly driven by Unified Robot Description Format (URDF) forward kinematics, and independent, manipulable object nodes.
  2. Task-Oriented Spatio-Temporal Alignment: Rather than relying on noisy optical flow or unconstrained deformation fields, the authors exploit the inherent structure of manipulation tasks. They segment the trajectory into Motion Phases (free-space transport) and Skill Phases (contact-rich interactions like grasping or placing). They then apply phase-specific geometric constraints: keeping objects static when untouched, rigidly attaching them to the gripper during transport, and using point tracking (via CoTracker) only during active manipulation.
  3. Topology-Preserving Data Augmentation: To scale up training data, ManiSplat employs a "skill-preserving, motion-adaptive" strategy. It applies random rigid transformations to the object and gripper during the critical skill phase (preserving local interaction mechanics) and automatically re-plans a collision-free path for the motion phases using a motion planning framework.
Detailed pipeline of ManiSplat including Gaussian Scene Graph, Phase Segmentation, and Trajectory Synthesis.

Datasets

The framework was evaluated using both simulation and real-world data:

  • RoboTwin: A simulation platform used to provide high-fidelity rendering and precise ground-truth parameters to quantitatively assess pose estimation and reconstruction accuracy.
  • Real-World Dataset: Captured using an AgileX dual-arm manipulator, recording complex interaction sequences (such as grasping a flower and inserting it into a vase) in physical environments.

Key Insights and Ideas

  • Domain Knowledge beats Pure Vision: Purely visual tracking methods tend to fail or over-smooth during sudden contact events like the exact moment of a grasp. Injecting robotic domain knowledge—specifically, the alternating structure of motion and skill phases—drastically stabilizes pose estimation.
  • Kinematics as a Strong Regularizer: By driving the robot's Gaussians directly through URDF kinematics, the system guarantees geometric consistency for the robot arm, allowing the optimization process to focus entirely on appearance alignment and object tracking.
  • Decoupling Enables Infinite Data: Because objects are represented as independent Gaussian subfields, they can be rearranged arbitrarily. This turns a single real-world video demonstration into a generator for infinite, photorealistic, and physically feasible training variations for downstream policy learning.

Key Results

  • Superior Reconstruction: ManiSplat significantly outperformed state-of-the-art dynamic baselines (like 4DGS and DeformGS). It achieved a Full Image PSNR of 32.31 and a massive jump in object-specific rendering quality (26.85 PSNR vs. HUGS's 18.90), effectively eliminating motion blur during rapid gripper movements.
  • Highly Accurate Tracking: On the RoboTwin benchmark, ManiSplat reduced the average translation error to 0.5864 cm and rotation error to 5.185°, outperforming the GenPose++ baseline (0.9965 cm, 7.109°).
  • Photorealistic Augmentation: Qualitatively, the pipeline successfully synthesized diverse, novel trajectory variations from a single source video while maintaining sharp boundaries and realistic lighting consistency.

Limitations and Future Work

Despite its strengths, ManiSplat's rendering quality can degrade when data augmentation introduces large spatial offsets far from the original viewpoints. This is a fundamental limitation of sparse-view 3DGS optimization, where Gaussians optimized from limited angles lack geometric consistency from unobserved perspectives.

To address this, the authors suggest two promising future directions:

  1. Replacing 3DGS with 2D Gaussian Splatting (2DGS) to enforce better surface consistency and view synthesis quality.
  2. Incorporating multi-demonstration fusion to combine visual information from multiple reference videos, building a more robust and complete scene representation.

Summary updated

Abstract

Reconstructing dynamic and interactive 3D scenes from real-world observations remains a fundamental challenge in computer vision and robotics. While recent advances in 3D Gaussian Splatting have enabled high-fidelity static reconstruction, extending it to interactive environments with articulated robots and manipulable objects remains difficult due to complex contact interactions and abrupt pose changes. To address these challenges, we introduce ManiSplat, a unified framework that reconstructs controllable and decoupled Gaussian digital twins directly from monocular ego-view robotic videos. Our method introduces a Graph-Structured Disentangled Representation that separates the robot, objects, and background into independently optimizable Gaussian subfields organized within a scene graph. To ensure stability, we propose a Task-Oriented Spatio-Temporal Alignment module that leverages the inherent logic of manipulation tasks-alternating between Motion and Skill phases-to construct accurate pseudo-ground-truth trajectories. Finally, a joint photometric-geometric optimization ensures the reconstructed scenes are temporally coherent, physically consistent, and simulation-ready. Extensive experiments demonstrate that our approach reconstructs interaction-driven dynamic scenes with high fidelity and controllability, effectively supporting downstream robotic tasks and policy learning.
BibTeX
@misc{hu2026manisplat,
  author = {Wenhao Hu and Haonan Zhou and Liu Liu and Yun Du and Xinjie Wang and Ziang Li and Zhizhong Su and Gaoang Wang},
  title = {{ManiSplat: Manipulation Trajectory Synthesis from Monocular Video via Decoupled 3D Gaussian Splatting}},
  year = {2026},
  eprint = {2606.10645},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.10645},
}

Loading related content...

Figure