TL;DR
The RoboCasa-Shift benchmark moves beyond simple "success or failure" metrics to provide a diagnostic look at how Vision-Language-Action (VLA) policies fail under distribution shifts. By using counterfactual pairing and stage-aware tracking, it reveals that aggregate success rates often mask significant episode-level sensitivity and that different perturbations cause failures at vastly different stages of a long-horizon task.
What Problem Does This Paper Address?
Evaluating the robustness of VLA models—like those used to control household robots—is typically done by measuring the aggregate success rate under various perturbations (e.g., changing the lighting or camera angle). However, the authors argue that this approach is a "blunt instrument" for two main reasons:
- Hidden Sensitivity: Aggregate metrics can hide episode-level changes. For instance, if a perturbation causes five previously successful runs to fail, but also causes five previously failing runs to succeed by chance, the aggregate success rate remains unchanged despite the policy being highly sensitive to the shift.
- Lack of Granularity: In long-horizon tasks (like washing a pan), a binary outcome doesn't tell you where the robot failed. Did it fail to reach for the pan, or did it fail at the very last step of turning off the water? Understanding the "semantic depth" of a failure is crucial for debugging and deployment.
Methods
The authors introduce three key components in the RoboCasa-Shift pipeline:
- Controlled Single-Factor Shifts: They evaluate four specific perturbations—Illumination, Camera Orientation, Robot Base Initialization, and Gaussian Blur—each at three calibrated severity levels ($L_1$ to $L_3$).
- Counterfactual Pairing: To isolate the effect of a perturbation, they compare a "clean" rollout with a "perturbed" rollout that shares the exact same task instance, initial simulator state, and policy random seed. This ensures that any change in outcome is due solely to the distribution shift, not environmental randomness.
- StageTracker (Stage-Aware Diagnosis): They represent long-horizon tasks as a sequence of semantic subgoals (e.g., $g_1$: reach, $g_2$: grasp, $g_3$: place). Using simulator-state predicates, they track how many subgoals the robot completes before failing, allowing them to localize failures in the execution timeline.
Datasets
The benchmark is built on RoboCasa365, a large-scale simulation framework for household manipulation.
- Task Suite: The authors screened 16 tasks and selected four "clean-qualified" tasks (PreSoakPan, RinseSinkBasin, StackBowlsCabinet, WashLettuce) where the base policy achieved at least a 60% success rate.
- Evaluated Policy: The primary evaluation was conducted on GR00T N1.5, a state-of-the-art VLA model. Other models like SmolVLA and $\pi_0$ were screened but failed to meet the initial "clean competence" threshold for these complex tasks.
Key Insights and Ideas
- Aggregate Metrics are Deceptive: The authors found cases where the change in success rate ($\Delta SR$) was zero, yet 10% of the episodes changed their individual outcomes. This "balanced" transition (Success $\rightarrow$ Failure vs. Failure $\rightarrow$ Success) proves that a stable aggregate success rate does not imply robustness.
- Non-Monotonic Degradation: Surprisingly, increasing the severity of a perturbation doesn't always lead to worse performance. For camera orientation, the policy actually performed better at $L_3$ (severe) than at $L_2$ (moderate) in some tasks, suggesting complex, non-linear failure modes in VLA models.
- Semantic Depth Matters: Two different perturbations might result in the same 20% drop in success rate, but one might cause the robot to fail immediately, while the other allows it to complete 90% of the task. Gaussian blur, for example, tended to cause much earlier failures as severity increased compared to robot base shifts.
Key Results
- Gaussian Blur was the most consistent "policy killer," with macro-averaged $\Delta SR$ reaching 0.738 at $L_3$ (total failure). It also showed the clearest trend of shifting failures to earlier stages of the task as severity increased.
- Camera Orientation and Robot Base shifts showed significant task-level heterogeneity. A shift that crippled the robot in one task (e.g., PreSoakPan) might have negligible impact on another (e.g., WashLettuce).
- Stage-Aware Analysis: The authors demonstrated that at $L_3$ Gaussian blur, 100% of failures occurred before the first semantic milestone was reached. In contrast, for camera orientation at $L_2$, only 19% of failures were "first-stage," with most occurring much later in the execution.
Limitations and Future Work
- Policy Coverage: Only one policy (GR00T N1.5) was robust enough on the base tasks to be fully evaluated. The authors note the need to extend this to a broader range of models as they become more capable.
- Sim-to-Real: All evaluations were performed in simulation. Validating whether these diagnostic patterns (like the non-monotonic response to camera shifts) hold true on physical hardware is a critical next step.
- Task Diversity: The current suite focuses on four specific household tasks. Expanding the "StageTracker" approach to a wider variety of task structures and mobility settings would improve the benchmark's generalizability.
Summary updated