Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos

Gengshan Yang, Andrea Bajcsy, Shunsuke Saito, Angjoo Kanazawa

TL;DR

Agent-to-Sim (ATS) is a novel framework that transforms a collection of casual smartphone videos—like a month’s worth of clips of your pet—into an interactive 3D behavior simulator. By building a persistent 4D representation across fragmented video clips, ATS learns how an agent interacts with its environment and the person filming it, enabling "real-to-sim" transfer without the need for expensive motion-capture studios.

The Challenge of Casual Video

Building realistic 3D behavior models for animals or humans usually requires invasive setups: marker-based suits, multi-view camera rigs, or controlled studio environments. While these produce high-quality data, they fail to capture how agents behave naturally in their everyday lives.

The alternative—learning from casual videos—is incredibly difficult. Most existing methods reconstruct each video clip independently. However, to learn behavior, a model needs persistence: it must understand that the cat on the rug in Video A is the same cat in the same room in Video B, recorded three weeks later. Mapping thousands of frames from different angles and times into one consistent 3D "world" is a massive registration challenge.

Mapping disparate video clips into a single, persistent 3D scene to track agent and observer trajectories over time.

How Agent-to-Sim Works

ATS tackles this challenge through a two-stage pipeline: 4D Reconstruction followed by Interactive Behavior Learning.

The Agent-to-Sim pipeline: from casual video collections to 4D reconstruction, behavior learning, and finally an interactive simulator.
  1. Coarse-to-Fine Registration: To align disparate video clips, the authors use "Neural Localization." They leverage the 3D awareness of foundation models (like DINO-v2) to regress initial camera poses relative to a template 3D mesh of the scene. They then refine this using a Feature-metric Loss, which uses differentiable rendering to align the 3D model with the 2D features of the video frames.
  2. Persistent 4D Representation: The world is factored into a static "canonical" scene and a deformable agent. Using a technique called Scene Annealing, the model ensures that while the scene can change slightly (e.g., a moved chair), the underlying structure remains consistent across all videos.
  3. Hierarchical Generative Modeling: Instead of predicting every joint movement at once, ATS uses a hierarchy of diffusion models:
  4. Goal Model: Predicts where the agent wants to go.
  5. Path Model: Plans the 3D trajectory to that goal.
  6. Body Pose Model: Generates the specific skeletal motion (using 25 "bones") to follow that path.
  7. Ego-Perception: The model is conditioned on what the agent "sees" (local 3D scene features) and the "observer's" (the person filming) movements, allowing the simulated agent to react when a human approaches.

Key Insights and Results

The core insight of ATS is that persistence is the bridge to behavior modeling. Behavior isn't just about motion; it's about the relationship between motion and the environment. By forcing all videos into one persistent 4D space, the model learns spatial priors—like the fact that cats prefer sitting on sofas rather than walking through walls. Furthermore, breaking the behavior generation into a Goal $\rightarrow$ Path $\rightarrow$ Pose hierarchy makes the diffusion process much more stable and allows the model to generalize better from limited data.

To validate the approach, the authors introduced a new longitudinal dataset captured via monocular RGBD (iPhone) videos, including a 23-video dataset of a cat captured over 37 days, alongside smaller collections for a human, dog, and bunny.

Results highlight: * Superior Registration: ATS achieved a rotation error of only 6.35°, compared to 59.19° for multi-video baselines (like TotalRecon), proving the effectiveness of their coarse-to-fine approach. * Interactive Realism: The resulting simulator generates plausible, novel behaviors. Even if the training data only had one clip of a cat jumping off a table, the model can generate variations where the cat jumps to the left, right, or center while avoiding obstacles. * Scene Awareness: The agent abides by 3D geometry. Without scene conditioning, the simulated agent might walk through walls; with it, the agent naturally navigates around furniture.

Limitations and Future Work

While ATS represents a significant step forward, it has limitations. The current model assumes the environment is mostly static and cannot handle articulated scene elements (like an agent opening a door). It also models kinematics rather than physics, which can occasionally lead to "floaty" movement or feet sliding. Finally, behavior is currently modeled in short ~6-second windows. Future work will look to incorporate memory modules for long-term behavior planning and scale the framework to handle multiple interacting agents.

Summary updated

Abstract

We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal agents non-invasively through video observations recorded over a long time-span (e.g. a month) in a single environment. Modeling 3D behavior of an agent requires persistent 3D tracking (e.g., knowing which point corresponds to which) over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on animals given monocular RGBD videos captured by a smartphone. Project page: gengshan-y.github.io/agent2sim-www.
BibTeX
@inproceedings{yang2025agent,
  author = {Gengshan Yang and Andrea Bajcsy and Shunsuke Saito and Angjoo Kanazawa},
  title = {{Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=y80D4IojuY},
}
Figure