TL;DR
This paper reveals that environmental light levels (lux) are a poor proxy for what a robot actually "sees." By testing four popular imitation-learning policies under an 11.5x drop in room illuminance, the authors found that automatic camera settings (exposure and white balance) almost entirely compensated for the darkness, leading to virtually no change in robot success rates.
What Problem Does This Paper Address?
In the world of real-robot benchmarks, researchers often record room illuminance in lux to document lighting conditions. The assumption is that if the lux level is the same, the visual input to the robot's policy is consistent. Conversely, a massive drop in lux is expected to significantly degrade performance, as seen in many simulated benchmarks.
However, this paper challenges that assumption. The authors argue that on typical low-cost robot rigs, the camera's internal processing pipeline—specifically automatic exposure and white balance—acts as a powerful "equalizer." This creates a gap between the physical environment and the digital image, meaning that lux alone does not describe the distribution shift a policy actually faces.
Methods
The researchers used a 6-DoF SO-101 follower arm running the LeRobot framework. They evaluated four widely-used imitation-learning policies:
- ACT (Action Chunking with Transformers)
- SmolVLA (a vision-language-action model)
- Diffusion Policy (DP)
- $\pi_{0.5}$ (a pretrained VLA model)
All models were trained on the same 50 teleoperated demonstrations of a pick-and-place task (picking a green ball while ignoring red distractors) recorded at ~500 lux.
The "stress test" involved running 80 rollouts: 10 per policy in "Normal" light (~520 lux) and 10 per policy in "Low" light (~45 lux, an 11.5x reduction). Crucially, the cameras (a Logitech Brio 300 and a generic wrist module) were left on automatic settings, mirroring common practice in many labs.
Datasets
- Training Data: 50 teleoperated demonstrations recorded at ~500 lux.
- Evaluation Data: 80 new rollouts (40 normal, 40 low-light).
- Public Release: All rollout datasets, scored logs, and analysis scripts are available on Hugging Face under
fysical/rollout_s2smoke_*.
Key Insights and Ideas
- The Camera Pipeline is a Filter: The most striking finding was that an 11.5x drop in room light only resulted in a 3% drop in mean image brightness for the front camera. The camera's automatic exposure simply "cranked up the gain," delivering images to the policy that looked nearly identical to the training data.
- Lux is Deceptive: Reporting lux in a paper without reporting camera settings (exposure time, gain, white balance) is scientifically incomplete. The robot's "eye" is not a passive sensor; it's an active processor.
- Sim vs. Real Discrepancy: In simulation, lighting changes usually modify pixel values directly. In reality, the hardware compensates before the policy ever sees a pixel. This suggests that some "robustness" results in literature might actually be measuring the camera's hardware robustness rather than the policy's algorithmic robustness.
- Single-Block Noise: An accidental repeat of one 10-rollout block yielded the same success count (7/10) but failed on different ball placements. This highlights that small-sample benchmarks (like 10 rollouts) are extremely noisy and should be interpreted with caution.
Key Results
The quantitative impact of the 11.5x light drop was negligible:
- Success Rate: 20/40 in normal light vs. 19/40 in low light.
- Effect Size: A statistically insignificant change of -2.5 percentage points.
- Image Statistics: While room light fell by over 1000%, the wrist camera images were only 19% darker, and the front camera was practically unchanged.
As shown in the visual comparison below, the images provided to the models in "low light" are barely distinguishable from the "normal" light training data.
Limitations and Future Work
- Sample Size: 10 rollouts per cell is a small sample, capable of detecting only very large performance drops.
- Confounding Factors: The low-light tests were conducted on a different day and in a different order, which could introduce minor environmental drifts.
- Lack of Manual Control: The authors did not explicitly test "locked" exposure settings to see how the policies would fail if the images actually became 11x darker.
- Scope: The study was limited to one specific task and one hardware rig. Future work should investigate if these findings hold for more complex visual scenes or different sensor types.
The authors conclude with a practical recommendation for the field: Report camera settings, not just lux, and lock exposure if you truly want to test a policy's robustness to lighting.
Summary updated