Rethinking Camera Choice: An Empirical Study on Fisheye Camera Properties in Robotic Manipulation

Han Xue, Nan Min, Xiaotong Liu, Wendi Chen, Yuan Fang, Jun Lv, Cewu Lu, Chuan WenarXiv 2026 · arXiv

TL;DR

This paper presents the first systematic empirical study on how wrist-mounted fisheye cameras affect imitation learning in robotic manipulation. While a wide Field of View (FoV) significantly boosts spatial localization and scene generalization, it requires visually rich environments and diverse training backgrounds to prevent overfitting. To solve the catastrophic failures that occur when transferring policies to new fisheye lenses, the authors introduce a simple yet highly effective Random Scale Augmentation (RSA) strategy.

What Problem Does This Paper Address?

There is a booming trend in robotics toward equipping robot wrists with fisheye cameras (as seen in recent high-profile projects like UMI and GEN-0). Their massive, often $>180^\circ$ Field of View provides a sweeping perspective of the workspace, which is incredibly useful for keeping both the gripper and target objects in sight.

However, our systematic understanding of how these lenses affect downstream policy learning has lagged behind their adoption. Fisheye lenses introduce severe radial distortion and alter object scales drastically depending on the camera's intrinsic parameters. This paper tackles three fundamental questions:

  1. Spatial Localization: Does a wider FoV actually help a policy reason about space?
  2. Scene Generalization: Do fisheye cameras make policies robust against novel or distracting backgrounds?
  3. Hardware Generalization: Can a policy trained on one fisheye camera successfully transfer to a completely different fisheye lens?
Figure 1. Overview of the four factors analyzed to address our Research Questions (RQs). We study: (a) Camera Model (fisheye vs. pinhole) as our primary comparison; (b) Scene Complexity (poor vs. rich) for spatial localization (RQ1); (c) Scene Diversity (1 vs. N scenes) for scene generalization (RQ2); and (d) Camera Parameters (varied intrinsics) for hardware generalization (RQ3).

Methods

To isolate the purely visual effects of camera choice, the authors designed a state-free (proprioception-free) visual imitation learning framework powered by Diffusion Policy. By removing internal joint and pose data, the policy is forced to rely entirely on visual cues for localization.

The methodology introduces two key technical components:

  • Fisheye Simulation in MuJoCo: Since standard physics engines lack native fisheye rendering, the authors built a stable, efficient two-stage pipeline. It captures a $360^\circ$ spherical view using six virtual cameras arranged as a cubemap, stitches them into an intermediate equirectangular image, and then projects it into a distorted fisheye view.
  • Random Scale Augmentation (RSA): To address cross-camera transfer failures, the authors introduced RSA. Instead of standard fixed-scale cropping, RSA samples a wide uniform distribution of scale factors ($s \in [0.7, 1.3]$), center-crops the image, and pads any "zoomed-out" borders with black pixels. This forces the network to learn relative spatial relationships rather than memorizing absolute pixel sizes.
Figure 2. The implementation pipeline of fisheye camera simula- tion in MuJoCo [48].

Datasets

The authors evaluated their approach across both simulated and real-world environments:

  • Simulation Benchmarks: Six challenging tasks adapted from Robomimic (Square, Tool Hang) and MimicGen (Coffee, Threading, Assembly, Mug Cleanup). Background textures were varied using the MimicLab asset library.
  • Real-World Setup: A Flexiv Rizon 4 robot arm performing three tasks: Pick Cup (spatial generalization), Fold Towel (deformable object manipulation), and Hang Chinese Knot (high-precision rotation). Backgrounds were altered using a variable system of patterned cloths.

Key Insights and Ideas

  • The "Visual Anchor" Dependency: A wide FoV only improves spatial localization if the environment is visually complex. In feature-poor, solid-colored environments, the fisheye's advantage completely vanishes because there are no static background features to serve as stable visual anchors.
  • The Double-Edged Sword of FoV: Fisheye cameras overfit to simple scenes much faster than pinhole cameras do. However, if you train them with sufficient background diversity, they unlock vastly superior zero-shot generalization to unseen environments.
  • The Depth-Scale Ambiguity: Naive cross-camera transfer between different fisheye lenses fails catastrophically due to "scale overfitting." When a policy encounters a new lens with a wider FoV, objects appear smaller. The policy misinterprets this visual scale change as a depth change, believing the object is farther away, which leads to severe overshooting or collisions.
Figure 3. Random Crop Augmentation (fixed scale) vs. Random Scale Augmentation (RSA) for cross-camera generalization.

Key Results

  • Real-World Localization: In feature-rich environments, switching from a poor background to a rich background yielded an average performance gain of +0.39 for fisheye policies, compared to just +0.18 for standard pinhole cameras.
  • Scene Generalization Scaling: In real-world testing, a fisheye policy's zero-shot success rate on entirely unseen backgrounds rapidly exceeded 95% when trained on just 8 diverse scenes.
  • Mitigating Hardware Shift: When transferring a policy trained on a $180^\circ$ lens to a wider $220^\circ$ lens, a standard baseline policy failed completely (score dropped to 0.0025). Equipping the policy with RSA successfully recovered performance to 0.6000 without any hardware-specific fine-tuning.

Limitations and Future Work

While the paper provides actionable guidelines, it notes a few limitations. The performance scaling curve in simulation was noticeably less steep than in the real world. The authors attribute this to the lower visual complexity of simulated backgrounds and the use of a non-pretrained ResNet-18 encoder in simulation versus a powerful, pretrained CLIP ViT encoder in the real world.

Future work includes extending this systematic analysis beyond imitation learning to Vision-Language-Action (VLA) models and reinforcement learning paradigms.

Summary updated

Abstract

The adoption of fisheye cameras in robotic manipulation, driven by their exceptionally wide Field of View (FoV), is rapidly outpacing a systematic understanding of their downstream effects on policy learning. This paper presents the first comprehensive empirical study to bridge this gap, rigorously analyzing the properties of wrist-mounted fisheye cameras for imitation learning. Through extensive experiments in both simulation and the real world, we investigate three critical research questions: spatial localization, scene generalization, and hardware generalization. Our investigation reveals that: (1) The wide FoV significantly enhances spatial localization, but this benefit is critically contingent on the visual complexity of the environment. (2) Fisheye-trained policies, while prone to overfitting in simple scenes, unlock superior scene generalization when trained with sufficient environmental diversity. (3) While naive cross-camera transfer leads to failures, we identify the root cause as scale overfitting and demonstrate that hardware generalization performance can be improved with a simple Random Scale Augmentation (RSA) strategy. Collectively, our findings provide concrete, actionable guidance for the large-scale collection and effective use of fisheye datasets in robotic learning. More results and videos are available on https://robo-fisheye.github.io/
BibTeX
@misc{xue2026rethinking,
  author = {Han Xue and Nan Min and Xiaotong Liu and Wendi Chen and Yuan Fang and Jun Lv and Cewu Lu and Chuan Wen},
  title = {{Rethinking Camera Choice: An Empirical Study on Fisheye Camera Properties in Robotic Manipulation}},
  year = {2026},
  eprint = {2603.02139},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02139},
}
Figure