TL;DR
Agent-to-Sim (ATS) is a novel framework that transforms a collection of casual smartphone videos—like a month’s worth of clips of your pet—into an interactive 3D behavior simulator. By building a persistent 4D representation across fragmented video clips, ATS learns how an agent interacts with its environment and the person filming it, enabling "real-to-sim" transfer without the need for expensive motion-capture studios.
The Challenge of Casual Video
Building realistic 3D behavior models for animals or humans usually requires invasive setups: marker-based suits, multi-view camera rigs, or controlled studio environments. While these produce high-quality data, they fail to capture how agents behave naturally in their everyday lives.
The alternative—learning from casual videos—is incredibly difficult. Most existing methods reconstruct each video clip independently. However, to learn behavior, a model needs persistence: it must understand that the cat on the rug in Video A is the same cat in the same room in Video B, recorded three weeks later. Mapping thousands of frames from different angles and times into one consistent 3D "world" is a massive registration challenge.
How Agent-to-Sim Works
ATS tackles this challenge through a two-stage pipeline: 4D Reconstruction followed by Interactive Behavior Learning.
- Coarse-to-Fine Registration: To align disparate video clips, the authors use "Neural Localization." They leverage the 3D awareness of foundation models (like DINO-v2) to regress initial camera poses relative to a template 3D mesh of the scene. They then refine this using a Feature-metric Loss, which uses differentiable rendering to align the 3D model with the 2D features of the video frames.
- Persistent 4D Representation: The world is factored into a static "canonical" scene and a deformable agent. Using a technique called Scene Annealing, the model ensures that while the scene can change slightly (e.g., a moved chair), the underlying structure remains consistent across all videos.
- Hierarchical Generative Modeling: Instead of predicting every joint movement at once, ATS uses a hierarchy of diffusion models:
- Goal Model: Predicts where the agent wants to go.
- Path Model: Plans the 3D trajectory to that goal.
- Body Pose Model: Generates the specific skeletal motion (using 25 "bones") to follow that path.
- Ego-Perception: The model is conditioned on what the agent "sees" (local 3D scene features) and the "observer's" (the person filming) movements, allowing the simulated agent to react when a human approaches.
Key Insights and Results
The core insight of ATS is that persistence is the bridge to behavior modeling. Behavior isn't just about motion; it's about the relationship between motion and the environment. By forcing all videos into one persistent 4D space, the model learns spatial priors—like the fact that cats prefer sitting on sofas rather than walking through walls. Furthermore, breaking the behavior generation into a Goal $\rightarrow$ Path $\rightarrow$ Pose hierarchy makes the diffusion process much more stable and allows the model to generalize better from limited data.
To validate the approach, the authors introduced a new longitudinal dataset captured via monocular RGBD (iPhone) videos, including a 23-video dataset of a cat captured over 37 days, alongside smaller collections for a human, dog, and bunny.
Results highlight: * Superior Registration: ATS achieved a rotation error of only 6.35°, compared to 59.19° for multi-video baselines (like TotalRecon), proving the effectiveness of their coarse-to-fine approach. * Interactive Realism: The resulting simulator generates plausible, novel behaviors. Even if the training data only had one clip of a cat jumping off a table, the model can generate variations where the cat jumps to the left, right, or center while avoiding obstacles. * Scene Awareness: The agent abides by 3D geometry. Without scene conditioning, the simulated agent might walk through walls; with it, the agent naturally navigates around furniture.
Limitations and Future Work
While ATS represents a significant step forward, it has limitations. The current model assumes the environment is mostly static and cannot handle articulated scene elements (like an agent opening a door). It also models kinematics rather than physics, which can occasionally lead to "floaty" movement or feet sliding. Finally, behavior is currently modeled in short ~6-second windows. Future work will look to incorporate memory modules for long-term behavior planning and scale the framework to handle multiple interacting agents.
Summary updated