Behavior Graphs: Unsupervised Graph Structures for Interpreting Robot Deployment Data

Erik Bauer, Jakob Thumm, Christopher Agia, Rohan Sinha, Chen Xu, Haruki Nishimura, Jeannette Bohg, Marco Pavone, Masha ItkinaarXiv 2026 · arXiv

TL;DR

Behavior Graphs provide an unsupervised way to compress thousands of high-dimensional robot rollouts into a single, interpretable directed graph. By clustering policy embeddings into semantic behavior modes, this approach allows researchers to pinpoint exactly where and why a policy fails, enabling more efficient data collection and state-of-the-art progress estimation without manual labeling.

What Problem Does This Paper Address?

The "data flywheel" is the holy grail of robot learning: you deploy a policy, observe its failures, collect new demonstrations to fix those specific failures, and retrain. However, as deployments scale, researchers are drowning in data. Watching thousands of hours of rollout videos to find failure patterns is tedious, subjective, and doesn't scale.

Without a structured way to represent behavior, identifying the "root cause" of a failure is difficult. Is the robot failing because it missed the initial grasp, or because it couldn't handle the lighting on the second shelf? Behavior Graphs aim to turn this unstructured "pile of videos" into a map that clearly shows the paths to success and the common detours into failure.

Methods

The authors propose a pipeline to construct these graphs without any human supervision:

  1. Embedding: They take the state-action pairs from rollouts and pass them through an embedding function—typically the observation encoder of the robot's own policy.
  2. Temporal Aggregation: To handle noise, they use a sliding window to average these embeddings over time.
  3. Clustering: They use UMAP for dimensionality reduction and K-means to group similar segments across all rollouts into $K$ distinct "behavior modes."
  4. Graph Construction: These modes become nodes. Directed edges are drawn based on the temporal order of behaviors in the rollouts, weighted by empirical transition probabilities (e.g., "30% of the time, 'Grasping the red can' leads to 'Stuck near the can'").
  5. VLM Captioning: To make the graph human-readable, they use a Vision-Language Model (VLM). The VLM looks at a "contact sheet" of images from a cluster and assigns a semantic label like "Pick up the pumpkin."
Fig. 1: How do robot policies fail during deployments, and which data should we collect to improve them? Raw policy rollouts that deployments produce are hard to interpret and offer limited insights. We introduce behavior graphs, which compress policy rollouts into an interpretable graph. Data collectors and researchers can use this structure to identify failure modes and guide targeted data collection, closing the policy improvement loop. Behavior graphs also enable state-of-the-art progress estimation without any additional model training.

Datasets

The researchers evaluated their approach across a massive suite of 43 datasets:

  • Simulation: Benchmarks from LIBERO, Robomimic, and MimicGen.
  • Real-World: Data collected on a DROID robot setup for three specific tasks: Set table (placing items from a shelf), Prepare tray (sliding a tray and placing bowls), and Cooking (a long-horizon task involving multiple ingredients and a pot).

Key Insights and Ideas

  • The Policy is its own Map: One of the most elegant ideas here is using the policy's own latent representations to define the graph. The robot’s internal "understanding" of the task naturally clusters into meaningful sub-behaviors.
  • Visualizing Failure Probabilities: By coloring nodes based on success likelihood (red to green) and sizing them by visitation frequency, the graph immediately surfaces "bottleneck" behaviors.
  • Zero-Shot Progress Estimation: Because the graph encodes the typical sequence of a task, it can be used as a progress estimator. By fitting a simple piecewise-linear model to the nodes, the authors achieved better results than specialized, supervised models.
  • Human-in-the-loop Efficiency: The user study revealed that while experts can reach the right conclusions by watching videos, they do so much faster and with significantly less mental effort when using the graph.
A behavior graph for a cooking task, showing various paths to success and failure points.

Key Results

  • User Preference: In a study with 18 participants, 16 preferred behavior graphs over traditional video browsers.
  • Reduced Cognitive Load: Using graphs reduced perceived temporal demand by 40.3% and total effort by 37.6% compared to watching videos.
  • Progress Prediction: On 43 datasets, behavior graphs improved the rank correlation of progress estimation by 22.9% and reduced Mean Absolute Error (MAE) by 53.5% relative to the strongest baseline (including VLM-based estimators like Robometer).
  • Ablation: The authors found that using the policy's own embeddings (rather than raw proprioception or DINOv3 features) led to the most accurate subtask segmentation.

Limitations and Future Work

  • Non-Causal Readout: The current progress estimation method is best suited for post-hoc analysis rather than real-time monitoring, as it relies on knowing which "segment" the robot is currently in.
  • Granularity Control: The number of clusters ($K$) is a hyperparameter. Setting it too low merges distinct behaviors; setting it too high makes the graph unreadable.
  • VLM Spatial Reasoning: While VLMs are good at captioning, they sometimes struggle with fine-grained spatial nuances (e.g., distinguishing a "near miss" from a "solid grasp"). The authors suggest that future improvements in VLM spatial understanding will make these graphs even more precise.

Overall, Behavior Graphs represent a significant step toward making robot "black box" policies transparent and manageable for the engineers tasked with improving them.

Summary updated

Abstract

Data flywheels promise continuous improvement via insights from deployment data. Understanding how the deployed model succeeds and fails can steer subsequent data collection towards patching failures and reinforcing successes. However, robot deployments create large volumes of high-dimensional rollout data, which is challenging to understand and confidently ground data collection decisions in. To make deployment data easier to analyze and act on, we propose behavior graphs, which compress policy rollouts into a single interpretable, directed graph. We construct the graphs by learning to identify similar behavior modes across rollouts, connecting the behaviors temporally, and finally linking them to outcomes. In a user study on three real-world tasks, where participants analyzed rollouts to choose a data collection strategy, 16 of 18 preferred behavior graphs to rollout videos, reaching comparable conclusions with 40.3% lower perceived temporal demand and 37.6% lower effort. Beyond interpretability for deployment data, behavior graphs are capable progress predictors: across 43 datasets, they improve the rank correlation between predicted progress and true temporal order by 22.9% and reduce mean absolute error by 53.5% relative to the strongest baseline. As the number of deployed robots grows, behavior graphs lend structure to high-dimensional policy rollouts and make deployment data easier to interpret and reason about.
BibTeX
@misc{bauer2026behavior,
  author = {Erik Bauer and Jakob Thumm and Christopher Agia and Rohan Sinha and Chen Xu and Haruki Nishimura and Jeannette Bohg and Marco Pavone and Masha Itkina},
  title = {{Behavior Graphs: Unsupervised Graph Structures for Interpreting Robot Deployment Data}},
  year = {2026},
  eprint = {2611.00405},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00405},
}
Figure