RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation

Haocheng Wang, Youquan Liu, Weiyu Guo, Aokai Zhang, Donghang Wu, Hongyu Liu, Sha Zhang, Weidong Yang, Ben FeiarXiv 2026 · arXiv

TL;DR

The RoboCasa-Shift benchmark moves beyond simple "success or failure" metrics to provide a diagnostic look at how Vision-Language-Action (VLA) policies fail under distribution shifts. By using counterfactual pairing and stage-aware tracking, it reveals that aggregate success rates often mask significant episode-level sensitivity and that different perturbations cause failures at vastly different stages of a long-horizon task.

What Problem Does This Paper Address?

Evaluating the robustness of VLA models—like those used to control household robots—is typically done by measuring the aggregate success rate under various perturbations (e.g., changing the lighting or camera angle). However, the authors argue that this approach is a "blunt instrument" for two main reasons:

  1. Hidden Sensitivity: Aggregate metrics can hide episode-level changes. For instance, if a perturbation causes five previously successful runs to fail, but also causes five previously failing runs to succeed by chance, the aggregate success rate remains unchanged despite the policy being highly sensitive to the shift.
  2. Lack of Granularity: In long-horizon tasks (like washing a pan), a binary outcome doesn't tell you where the robot failed. Did it fail to reach for the pan, or did it fail at the very last step of turning off the water? Understanding the "semantic depth" of a failure is crucial for debugging and deployment.

Methods

The authors introduce three key components in the RoboCasa-Shift pipeline:

  • Controlled Single-Factor Shifts: They evaluate four specific perturbations—Illumination, Camera Orientation, Robot Base Initialization, and Gaussian Blur—each at three calibrated severity levels ($L_1$ to $L_3$).
  • Counterfactual Pairing: To isolate the effect of a perturbation, they compare a "clean" rollout with a "perturbed" rollout that shares the exact same task instance, initial simulator state, and policy random seed. This ensures that any change in outcome is due solely to the distribution shift, not environmental randomness.
  • StageTracker (Stage-Aware Diagnosis): They represent long-horizon tasks as a sequence of semantic subgoals (e.g., $g_1$: reach, $g_2$: grasp, $g_3$: place). Using simulator-state predicates, they track how many subgoals the robot completes before failing, allowing them to localize failures in the execution timeline.
Fig. 2: Pipeline of RoboCasa-Shift. (a) Task Suite. Candi- date RoboCasa tasks are filtered through clean-competence screening to construct the formal evaluation suite. (b) Con- trolled Shifts. Selected tasks are evaluated under four single- factor distribution shifts with calibrated and frozen severity levels. (c) Counterfactual Pairing. Clean and perturbed roll- outs share the underlying task conditions and policy seed and differ only in the target shift. (d) Diagnosis. RoboCasa’s offi- cial task outcome is complemented by stage-aware semantic diagnostics of execution progress, first incomplete subgoal, and subgoal reach.

Datasets

The benchmark is built on RoboCasa365, a large-scale simulation framework for household manipulation.

  • Task Suite: The authors screened 16 tasks and selected four "clean-qualified" tasks (PreSoakPan, RinseSinkBasin, StackBowlsCabinet, WashLettuce) where the base policy achieved at least a 60% success rate.
  • Evaluated Policy: The primary evaluation was conducted on GR00T N1.5, a state-of-the-art VLA model. Other models like SmolVLA and $\pi_0$ were screened but failed to meet the initial "clean competence" threshold for these complex tasks.

Key Insights and Ideas

  • Aggregate Metrics are Deceptive: The authors found cases where the change in success rate ($\Delta SR$) was zero, yet 10% of the episodes changed their individual outcomes. This "balanced" transition (Success $\rightarrow$ Failure vs. Failure $\rightarrow$ Success) proves that a stable aggregate success rate does not imply robustness.
  • Non-Monotonic Degradation: Surprisingly, increasing the severity of a perturbation doesn't always lead to worse performance. For camera orientation, the policy actually performed better at $L_3$ (severe) than at $L_2$ (moderate) in some tasks, suggesting complex, non-linear failure modes in VLA models.
  • Semantic Depth Matters: Two different perturbations might result in the same 20% drop in success rate, but one might cause the robot to fail immediately, while the other allows it to complete 90% of the task. Gaussian blur, for example, tended to cause much earlier failures as severity increased compared to robot base shifts.
Fig. 4: End-to-end robustness under controlled distribu- tion shifts. Macro-averaged paired success-rate degradation ∆SR is shown for the four distribution-shift factors, with each factor evaluated at three frozen severity levels (L1– L3). Positive values indicate degradation relative to the matched clean condition. Error bars denote 95% confidence intervals obtained from paired bootstrap resampling. Severity levels are ordinal within each perturbation factor, and equal level indices across different factors do not imply physically equivalent perturbation magnitudes.

Key Results

  • Gaussian Blur was the most consistent "policy killer," with macro-averaged $\Delta SR$ reaching 0.738 at $L_3$ (total failure). It also showed the clearest trend of shifting failures to earlier stages of the task as severity increased.
  • Camera Orientation and Robot Base shifts showed significant task-level heterogeneity. A shift that crippled the robot in one task (e.g., PreSoakPan) might have negligible impact on another (e.g., WashLettuce).
  • Stage-Aware Analysis: The authors demonstrated that at $L_3$ Gaussian blur, 100% of failures occurred before the first semantic milestone was reached. In contrast, for camera orientation at $L_2$, only 19% of failures were "first-stage," with most occurring much later in the execution.

Limitations and Future Work

  • Policy Coverage: Only one policy (GR00T N1.5) was robust enough on the base tasks to be fully evaluated. The authors note the need to extend this to a broader range of models as they become more capable.
  • Sim-to-Real: All evaluations were performed in simulation. Validating whether these diagnostic patterns (like the non-monotonic response to camera shifts) hold true on physical hardware is a critical next step.
  • Task Diversity: The current suite focuses on four specific household tasks. Expanding the "StageTracker" approach to a wider variety of task structures and mobility settings would improve the benchmark's generalizability.

Summary updated

Abstract

Vision-language-action (VLA) robustness is commonly summarized by aggregate task success, which obscures how individual executions change under distribution shifts and where failures emerge in long-horizon tasks. We introduce \textbf{RoboCasa-Shift}, a diagnostic robustness benchmark built on RoboCasa365 that combines controlled single-factor shifts, matched clean-perturbed rollouts in which each pair shares the same underlying settings and differs only in the target shift, and simulator-grounded stage-aware diagnostics. We evaluate GR00T N1.5 on four long-horizon tasks that pass a predefined clean-performance gate under four perturbation factors and three calibrated severity levels, including conditions that drive the evaluated policy to complete task failure, totaling 1,040 formal rollouts. Robustness responses vary substantially across factors and tasks, and increasing severity does not always produce monotonic degradation. For example, one camera condition yields $Δ\mathrm{SR}=0$ despite 8 of 80 matched episodes changing outcome. Stage-aware analysis further shows that the same end-to-end degradation can correspond to different execution-depth profiles. Camera $L_2$ and Gaussian blur $L_1$ both yield $Δ\mathrm{SR}=0.188$, yet their first-stage failure rates are 19.0\% and 36.4\%, respectively. Our findings suggest that VLA robustness should be viewed as a multi-dimensional property of execution rather than a single end-to-end performance measure, motivating evaluation protocols that jointly capture outcome degradation, paired outcome sensitivity, and failure progression. The code is publicly available at https://github.com/HaochengWang23/RoboCasa-Shift.
BibTeX
@misc{wang2026robocasa,
  author = {Haocheng Wang and Youquan Liu and Weiyu Guo and Aokai Zhang and Donghang Wu and Hongyu Liu and Sha Zhang and Weidong Yang and Ben Fei},
  title = {{RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation}},
  year = {2026},
  eprint = {2610.27861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.27861},
}
Figure