When Does Cross-View Consistency Matter in Multi-View Appearance Augmentation? A Controlled Study for Visuomotor Policies

Zixuan JiangarXiv 2026 · arXiv

In the quest to train robust robot "brains" (visuomotor policies), researchers often use generative AI to augment training data—changing backgrounds, lighting, or object colors in recorded demonstrations. For robots with multiple cameras (like a third-person view and a wrist-mounted view), a common assumption is that these augmented views must be perfectly consistent: if the block is red in one camera, it must be red in the other.

But is this "cross-view consistency" actually necessary for the robot to learn? A new study from Peking University suggests that for many visual factors, the answer is a surprising "no."

TL;DR

This paper demonstrates that while geometric consistency (where things are) is critical for multi-view robot control, appearance consistency (what things look like) can often be relaxed. By isolating "coupling" from "diversity," the authors show that policies can handle inconsistent colors and textures across views, provided the underlying task geometry remains sound.

What Problem Does This Paper Address?

Generative visual augmentation is a powerful tool for scaling robot learning without collecting new physical data. However, generating multi-view data that is perfectly consistent across cameras is computationally expensive and technically challenging.

The core question is: Does stronger cross-view consistency necessarily improve downstream control? If a robot can learn just as well from "inconsistent" views (e.g., a blue table in the third-person view but a green table in the wrist view), we can use simpler, faster, independent augmentation techniques. The authors seek to identify which specific forms of consistency matter and which are optional.

Methods

The researchers developed a controlled simulation design using simulation-state replay. Instead of just editing images, they replayed recorded robot states in a simulator and rendered each camera view separately under different "recipes."

Figure 1: Controlled visual augmentation. (a) State replay preserves physical trajectories and action labels while rendering each camera under its assigned recipe. (b) Shared and independent joint distributions have the same per-camera marginals. (c) Same-state examples of shared appearance (C1) and independent recoloring (C10), from spatial_8, demo 0, frame 0, seed 1, variant 1. The historical renders are rotated into policy-input orientation. The matched recoloring experiment compares shared recoloring (C9) with C10.

The key innovation is the use of marginal-matched distributions. They ensured that each individual camera received the same variety of perturbations (the "marginals") regardless of whether those perturbations were "coupled" (consistent across views) or "independent" (inconsistent). This allowed them to isolate the specific effect of cross-view coupling. They tested several types of consistency:

  • Global Appearance: Textures and lighting.
  • Ghost Injection: Adding visual distractors that don't exist in the physical simulation.
  • Target Color: Changing the color of the object being manipulated.
  • Geometry: Shifting the rendered position of the target object relative to its physical location.

Datasets

The study utilized seven manipulation tasks from the LIBERO benchmark, ranging from picking up bowls to opening drawers. They evaluated two popular policy architectures:

  1. Diffusion Policy (DP): A late-fusion model using ResNet18 encoders.
  2. Action Chunking Transformer (ACT): A transformer-based model that combines visual tokens.

Key Insights and Ideas

  • Appearance Invariance is Robust: For the Diffusion Policy, having inconsistent textures or target colors across views resulted in almost zero performance loss. The policy learns to be "invariant" to these changes as long as they are randomized during training.
  • Geometry is Non-Negotiable: Unlike color, geometric consistency is vital. When the target object's position was shifted inconsistently between the two views (e.g., shifted left in one, right in the other), success rates dropped significantly.
  • The "Where" vs. the "What": The results suggest that late-fusion policies rely on cameras to provide independent geometric cues. If those cues contradict each other, the policy fails. However, as long as the "where" is consistent, the "what" (color/texture) can vary.
  • Architecture Sensitivity: Interestingly, the ACT model showed a higher sensitivity to target-color inconsistency than the Diffusion Policy, suggesting that how a model fuses information (attention vs. concatenation) changes its reliance on consistency.

Key Results

  • Appearance Costs: For DP, the cost of moving from shared (consistent) to independent (inconsistent) appearance was negligible, with point estimates near zero. The largest upper bound of the 90% credible interval was only 2.55 percentage points.
  • Geometry Costs: "Anti-coupled" geometric observations (where views are shifted in different directions) reduced success by 3.30 percentage points compared to coupled observations.
  • Wrist-View Importance: Policies are heavily reliant on the wrist camera. Masking the wrist camera entirely led to a massive success rate drop of ~56 percentage points.
  • Feature Analysis: Feature invariance ratios showed that training with randomized appearance successfully "squashed" the model's sensitivity to those factors, regardless of whether the randomization was consistent across views.
Comparison of success-rate differences for various appearance and photometric jitter contrasts.

Limitations and Future Work

The authors acknowledge several boundaries to their findings:

  • Task Scope: The study was limited to seven simulation tasks and specific architectures (ResNet18/DP/ACT). Results might differ for massive Vision-Language-Action (VLA) models or real-world robots.
  • Low-Dose "Ghosts": The visual distractors (ghosts) used had low co-visibility (they weren't often seen by both cameras at once), which might explain why their consistency didn't matter.
  • Out-of-Distribution: The camera masking experiments are "out-of-distribution" interventions, which are useful for probing but don't perfectly reflect training-time behavior.

Future Work: The findings motivate a "selective consistency" approach to augmentation—prioritizing geometric fidelity and spatial correspondence while allowing for cheaper, independent randomization of background textures and lighting.

Summary updated

Abstract

Multi-view visual augmentation often seeks consistency across cameras, but it is unclear which forms of consistency matter for downstream control. We use simulation-state replay to manipulate cross-view coupling while preserving the per-camera perturbation distributions, physical trajectories, and action labels. Across seven LIBERO manipulation tasks, marginal-matched changes in global appearance, ghost injection, and target color have small average costs for a late-fusion Diffusion Policy: point estimates are near zero or negative, and the largest upper endpoint of the 90% posterior credible intervals is 2.55 percentage points. In contrast, geometrically anti-coupled observations reduce success by 3.30 percentage points relative to coupled observations (90% CrI [0.64, 6.28]), with displacement magnitude, direction marginals, and the number of affected views matched. Camera interventions reveal substantial dependence on wrist-view geometry, while feature analysis shows appearance sensitivity patterns associated with the factors randomized during training. ACT shows a higher target-color cost estimate than DP, with uncertainty that includes zero and sensitivity to task composition. Within the tested tasks and policy configurations, these findings motivate selective consistency constraints: appearance coupling can sometimes be relaxed, whereas task-relevant geometry and cross-view spatial correspondence warrant protection.
BibTeX
@misc{jiang2026when,
  author = {Zixuan Jiang},
  title = {{When Does Cross-View Consistency Matter in Multi-View Appearance Augmentation? A Controlled Study for Visuomotor Policies}},
  year = {2026},
  eprint = {2611.17751},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17751},
}
Figure