TL;DR
This paper presents the first systematic empirical study on how wrist-mounted fisheye cameras affect imitation learning in robotic manipulation. While a wide Field of View (FoV) significantly boosts spatial localization and scene generalization, it requires visually rich environments and diverse training backgrounds to prevent overfitting. To solve the catastrophic failures that occur when transferring policies to new fisheye lenses, the authors introduce a simple yet highly effective Random Scale Augmentation (RSA) strategy.
What Problem Does This Paper Address?
There is a booming trend in robotics toward equipping robot wrists with fisheye cameras (as seen in recent high-profile projects like UMI and GEN-0). Their massive, often $>180^\circ$ Field of View provides a sweeping perspective of the workspace, which is incredibly useful for keeping both the gripper and target objects in sight.
However, our systematic understanding of how these lenses affect downstream policy learning has lagged behind their adoption. Fisheye lenses introduce severe radial distortion and alter object scales drastically depending on the camera's intrinsic parameters. This paper tackles three fundamental questions:
- Spatial Localization: Does a wider FoV actually help a policy reason about space?
- Scene Generalization: Do fisheye cameras make policies robust against novel or distracting backgrounds?
- Hardware Generalization: Can a policy trained on one fisheye camera successfully transfer to a completely different fisheye lens?
Methods
To isolate the purely visual effects of camera choice, the authors designed a state-free (proprioception-free) visual imitation learning framework powered by Diffusion Policy. By removing internal joint and pose data, the policy is forced to rely entirely on visual cues for localization.
The methodology introduces two key technical components:
- Fisheye Simulation in MuJoCo: Since standard physics engines lack native fisheye rendering, the authors built a stable, efficient two-stage pipeline. It captures a $360^\circ$ spherical view using six virtual cameras arranged as a cubemap, stitches them into an intermediate equirectangular image, and then projects it into a distorted fisheye view.
- Random Scale Augmentation (RSA): To address cross-camera transfer failures, the authors introduced RSA. Instead of standard fixed-scale cropping, RSA samples a wide uniform distribution of scale factors ($s \in [0.7, 1.3]$), center-crops the image, and pads any "zoomed-out" borders with black pixels. This forces the network to learn relative spatial relationships rather than memorizing absolute pixel sizes.
Datasets
The authors evaluated their approach across both simulated and real-world environments:
- Simulation Benchmarks: Six challenging tasks adapted from Robomimic (Square, Tool Hang) and MimicGen (Coffee, Threading, Assembly, Mug Cleanup). Background textures were varied using the MimicLab asset library.
- Real-World Setup: A Flexiv Rizon 4 robot arm performing three tasks: Pick Cup (spatial generalization), Fold Towel (deformable object manipulation), and Hang Chinese Knot (high-precision rotation). Backgrounds were altered using a variable system of patterned cloths.
Key Insights and Ideas
- The "Visual Anchor" Dependency: A wide FoV only improves spatial localization if the environment is visually complex. In feature-poor, solid-colored environments, the fisheye's advantage completely vanishes because there are no static background features to serve as stable visual anchors.
- The Double-Edged Sword of FoV: Fisheye cameras overfit to simple scenes much faster than pinhole cameras do. However, if you train them with sufficient background diversity, they unlock vastly superior zero-shot generalization to unseen environments.
- The Depth-Scale Ambiguity: Naive cross-camera transfer between different fisheye lenses fails catastrophically due to "scale overfitting." When a policy encounters a new lens with a wider FoV, objects appear smaller. The policy misinterprets this visual scale change as a depth change, believing the object is farther away, which leads to severe overshooting or collisions.
Key Results
- Real-World Localization: In feature-rich environments, switching from a poor background to a rich background yielded an average performance gain of +0.39 for fisheye policies, compared to just +0.18 for standard pinhole cameras.
- Scene Generalization Scaling: In real-world testing, a fisheye policy's zero-shot success rate on entirely unseen backgrounds rapidly exceeded 95% when trained on just 8 diverse scenes.
- Mitigating Hardware Shift: When transferring a policy trained on a $180^\circ$ lens to a wider $220^\circ$ lens, a standard baseline policy failed completely (score dropped to 0.0025). Equipping the policy with RSA successfully recovered performance to 0.6000 without any hardware-specific fine-tuning.
Limitations and Future Work
While the paper provides actionable guidelines, it notes a few limitations. The performance scaling curve in simulation was noticeably less steep than in the real world. The authors attribute this to the lower visual complexity of simulated backgrounds and the use of a non-pretrained ResNet-18 encoder in simulation versus a powerful, pretrained CLIP ViT encoder in the real world.
Future work includes extending this systematic analysis beyond imitation learning to Vision-Language-Action (VLA) models and reinforcement learning paradigms.
Summary updated