FlowEval: How Much Adaptation Does an Embodied Foundation Model Need at Deployment?

Zhang Xi-Jia, Roman Yakunin, Shivang Chopra, Chengyue Huang, Che Chen, Siddharth Karamcheti, Zsolt KiraarXiv 2026 · arXiv

TL;DR

FlowEval is a training-free diagnostic tool that uses a handful of successful demonstrations to predict how much adaptation an embodied foundation model (EFM) needs for a new deployment. By measuring "predictive disagreement" through flow-matching metrics, it can distinguish between environments the model has already "seen" and those it hasn't, and it correlates strongly with the performance gains expected from future finetuning.

What Problem Does This Paper Address?

Large-scale pretraining has birthed powerful embodied foundation models (like Pi-0, GR00T, and OpenVLA) that can generalize across robots and tasks. However, "generalization" doesn't mean a model is ready for every deployment out of the box. Often, a model needs to be adapted (finetuned) to a specific robot's action space or a new environment's quirks.

The challenge for engineers is a "blind deployment" problem: How do you know if a pretrained model is a good fit for your specific task before you spend days and thousands of dollars finetuning it? Poor zero-shot performance is a noisy signal—it doesn't tell you if the model just needs 50 more demonstrations to "click" or if it requires 5,000. FlowEval aims to provide a lightweight "pre-flight check" to estimate this adaptation gap using only a small set of successful target-domain trajectories.

Methods

FlowEval leverages the internal mechanics of Flow Matching, a popular training objective for generative models in robotics. Unlike standard behavior cloning which just predicts an action, flow matching learns a vector field (a "flow") that pushes noisy inputs toward the ground-truth action.

The authors introduce two primary metrics:

  1. Velocity Error: The L2 distance between the model's predicted "velocity" (direction of improvement) and the reference velocity derived from a successful demonstration.
  2. Cosine Similarity: The directional agreement between these two vectors.

A key technical insight is that FlowEval initializes the action trajectory from pure Gaussian noise rather than the standard practice of starting from ground-truth-interpolated noisy actions. They found that starting from pure noise provides a much more stable and "cleaner" evaluation signal for distinguishing between tasks. Furthermore, they discovered that computing these metrics for just the first diffusion step is sufficient, making the evaluation extremely computationally efficient.

Datasets

The researchers evaluated FlowEval across four major model families: Pi-0.5, MolmoAct2, GR00T, and DreamZero. They tested these models on a diverse array of datasets spanning real-world and simulated manipulation:

  • Internet/Public Datasets: DROID, RoboSet, LIBERO, and MolmoAct2-Bimanual-YAM.
  • Simulation Benchmarks: RoboCasa365 (used for extensive finetuning experiments across 17 manipulation tasks).
  • Custom Data: Human-teleoperated demonstrations on Franka and OpenArm robot platforms.

Key Insights and Ideas

  • Training-Free Diagnostics: You can predict the "trainability" of a model on a new task without actually running a single gradient step. This allows practitioners to rank candidate models for a specific deployment based on their "fit."
  • The "Flow" as a Proxy for Familiarity: Because the flow-matching objective probes the model's local direction of improvement, it acts as a sensitive detector for whether a trajectory aligns with the model's learned prior.
  • Pure Noise > Noisy Actions: Counter-intuitively, the model's response to pure noise is more informative for evaluation than its response to a slightly corrupted version of the ground truth. This is likely because pure noise forces the model to rely entirely on its learned global structure.
  • Predicting the Scaling Law: FlowEval doesn't just say "this model is bad at this task." It correlates with the rate of improvement. If a model has high predictive agreement (high cosine similarity) on a few demos, it is likely to see much larger success rate gains as you provide more adaptation data.

Key Results

The authors demonstrated that FlowEval effectively separates "seen" from "unseen" deployments. As shown in Figure 1, trajectories from datasets included in a model's pretraining (Seen) cluster significantly closer to zero velocity error and unit cosine similarity than those from unseen distributions.

FlowEval results across various models and action representations.

More importantly, FlowEval predicts future performance. In finetuning experiments on 17 RoboCasa tasks, the initial FlowEval scores correlated with how the success rate improved as data scaled from 50 to 500 demonstrations (Figure 2). This suggests that the metric captures the "readiness" of the model's internal representations for a specific task.

Regression between success rate and metric values.

Limitations and Future Work

  • Action Space Mismatch: While FlowEval can be computed between different action representations (e.g., a model trained on joint positions evaluated on Cartesian data), this mismatch introduces significant noise that can swamp the "fit" signal.
  • Post-hoc Reconstructed Demos: The metric fails if demonstrations are mathematically reconstructed after the fact (e.g., converting Cartesian logs to joint space) rather than being the original control signals issued by a human. As seen in Figure 6, such reconstructions can "collapse" toward stationary states that confuse the metric.
FlowEval results on reconstructed Franka dataset showing failure modes.
  • Model Specificity: Currently, FlowEval is only applicable to models using flow-matching objectives. Future work could explore similar "predictive disagreement" metrics for other architectures like Transformers or Diffusion Policies.
  • Calibrated Predictions: The authors suggest that future versions could provide "calibrated" predictions—literally telling a user, "You will need approximately 250 demonstrations to reach an 80% success rate on this task."

Summary updated

Abstract

Large-scale pretraining is reducing the amount of adaptation required to deploy embodied foundation models, but it is not guaranteed that a pretrained model already provide the level of performance required in every deployment. Practitioners therefore need to understand how well a pretrained model matches a target deployment and how much adaptation data is left to be collected. We study whether a small set of successful target-domain demonstrations can reveal whether and how much target-domain data will be required before any adaptation. We introduce FlowEval, a set of training-free metrics that use the flow-matching objective to probe an embodied foundation model with a set of demonstrations. FlowEval quantifies predictive disagreement on successful target demonstrations to assess how well a pretrained model fits a target deployment. Across four embodied foundation model families: Pi-0.5, MolmoAct2, GR00T, and DreamZero, FlowEval distinguishes seen from unseen deployments across real-world and simulated manipulation. Further, by finetuning Pi-0.5 and GR00T on 17 RoboCasa365 manipulation tasks, we showed that FlowEval correlates with how success rate improves as additional adaptation data is provided. Together, these results suggest that FlowEval can serve as a lightweight, training-free diagnostic of how much model--deployment adaptation may be required.
BibTeX
@misc{xijia2026floweval,
  author = {Zhang Xi-Jia and Roman Yakunin and Shivang Chopra and Chengyue Huang and Che Chen and Siddharth Karamcheti and Zsolt Kira},
  title = {{FlowEval: How Much Adaptation Does an Embodied Foundation Model Need at Deployment?}},
  year = {2026},
  eprint = {2611.11650},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11650},
}
Figure