Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos

Gengshan Yang, Andrea Bajcsy, Shunsuke Saito, Angjoo Kanazawa

Abstract

We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal and human agents non-invasively through video observations recorded over a long time-span (e.g., a month) in a single environment. Modeling 3D behavior of an agent requires persistent 3D tracking over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on pets (e.g., cat, dog, bunny) and humans given monocular RGBD videos captured by a smartphone.

Introduction

Learning how agents—such as pets or humans—interact with their environment typically requires controlled motion-capture (MoCap) studios or massive amounts of manually labeled data. While studio data is high-quality, it often lacks the "naturalness" of everyday behavior. Conversely, casual "in-the-wild" video is natural but difficult to reconstruct with the temporal and 3D consistency required for behavior modeling.

The primary technical challenge is longitudinal consistency. Existing 4D reconstruction methods often fail when applied to collections of short clips taken over weeks or months, as lighting changes, furniture shifts, and camera viewpoints vary. ATS addresses this gap by building a persistent 4D representation from thousands of frames of casual video, enabling the training of generative behavior models that can predict plausible future actions based on past trajectories.

ATS models the behavior of agents by predicting plausible future paths based on past trajectories and environmental context.

Methodology

The ATS framework consists of two main stages: Coarse-to-Fine 4D Reconstruction and Hierarchical Behavior Learning.

1. Coarse-to-Fine 4D Reconstruction

To align videos taken at different times, ATS factorizes the world into a Canonical Structure (a time-independent neural field) and a Time-varying Structure (camera poses and agent deformations).

  • Neural Localization: Instead of traditional geometric matching, ATS uses a ResNet-18 to regress camera poses from DINOv2 features, providing robustness to lighting and scene shifts.
  • Deformable Agent Model: The agent is represented by 25 "bones" with learned skinning weights, allowing for complex non-rigid motions like stretching or walking.
  • Scene Annealing: To ensure a consistent global map, the model randomly swaps "scene codes" between videos during optimization, forcing the sharing of structural information across the entire collection.

2. Hierarchical Behavior Generation

Once 3D trajectories are extracted, ATS trains a generative model using a Hierarchical Diffusion approach. The model breaks behavior into three distinct steps: Goal Generation, Trajectory Generation, and Body Pose Generation. These are conditioned on Ego-perception (the agent’s view), the observer’s movement, and the agent's own past motion.

Results and Interactive Simulation

ATS was evaluated on a new dataset of monocular RGBD videos featuring cats, dogs, bunnies, and humans. The "Cat" dataset, for instance, included 23 clips recorded over 37 days. ATS achieved a rotation error of only 6.35°, significantly outperforming baselines like TotalRecon (59.19°) in longitudinal registration.

Beyond reconstruction, the framework enables realistic interactive simulations. By encoding the world from the agent's perspective, the model learns behaviors that generalize to new locations. In interactive tests, the generated agent realistically reacts to user inputs—for example, diverting its path when a user "blocks the way" or running away when approached too quickly.

The ATS pipeline processes longitudinal video clips to reconstruct a 4D scene and train a behavior model capable of reacting to user-defined scenarios like "blocking the way".

Conclusion and Future Work

ATS demonstrates that casual, longitudinal videos can be transformed into high-fidelity interactive simulators. While the current model assumes a mostly rigid background and handles single agents, future work will explore transient object handling and multi-agent interactions. Additionally, integrating physics-based constraints could further reduce artifacts like foot sliding, while memory modules could enable longer-term behavioral reasoning.

Summary updated

Abstract

We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal and human agents non-invasively through video observations recorded over a long time-span (e.g., a month) in a single environment. Modeling 3D behavior of an agent requires persistent 3D tracking (e.g., knowing which point corresponds to which) over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos captured by a smartphone.
BibTeX
@inproceedings{yang2025agent,
  author = {Gengshan Yang and Andrea Bajcsy and Shunsuke Saito and Angjoo Kanazawa},
  title = {{Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=y80D4IojuY},
}
Figure