Robots trained entirely in simulation usually learn to ignore unrealistic graphics because their training images are made wildly varied: random textures, random lighting, random camera poses. This is called domain randomization. One part of the image is almost never varied in this way, and that is the camera sensor itself. Real cameras count photons. In dim light they produce grain that grows with brightness, add an electronic noise floor that does not, carry pixel-to-pixel gain differences, and sometimes emit dead or hot pixels. A rendered image has none of this. We propose to randomize over a physically structured model of these effects during simulation training, sampling a different plausible camera for every episode from a prior fitted to public denoising datasets rather than to the deployment camera. We then ask whether the resulting manipulation policies transfer zero-shot to real cameras they have never seen, especially at low light, where the grain is strongest.
The claim is narrow and testable. Signal-dependent mixed noise is not interchangeable with the additive Gaussian jitter that vision pipelines already apply. A policy trained on the structured version should keep its real-world success rate as illumination drops. A policy trained on standard texture-and-lighting randomization, or on Gaussian noise with the same overall variance, should degrade. If matched-variance Gaussian noise does just as well, the physics is unnecessary, and the experiments are built so that this outcome would be visible.
Core Claim and Relation to Prior Work
The mixed-noise literature has worked out how real sensor noise is composed. Photon shot noise (Poisson) is paired with Gaussian read noise in fluorescence and light-sheet imaging (A convex 3D deconvolution algorithm for low photon count fluorescence imaging, Image Reconstruction in Light-Sheet Microscopy: Spatially Varying Deconvolution and Mixed Noise). Gaussian noise is paired with impulse spikes in denoising (Hybrid Model and Split Bregman Iteration Algorithm for Image Denoising, Infimal Convolution of Data Discrepancies for Mixed Noise Removal). These works also show that a fidelity term matched to only one component either smears spikes or leaves grain behind. That motivates us. It suggests that a vision encoder trained on the wrong noise family learns the wrong invariances. These works aim to invert noise with iterative solvers. We only need to sample it, which costs almost nothing at simulation frame rates.
The closest work is Sensor Transfer. It learns the parameters of a differentiable chain of chromatic aberration, blur, exposure, Poisson-Gaussian noise and color cast so that synthetic driving images match a specific target dataset's style. This raises car-detection average precision on Cityscapes from 30.13 to 35.48. It explicitly argues against randomization and requires target-domain images. It is evaluated on static detection, not on closed-loop control. Learning Camera-Aware Noise Models shows that real noise is a camera-specific fingerprint, with shot noise, dark current, stripes and blotches. It generates noise conditioned on an embedding extracted from noisy images of the target camera, again requiring target samples. Our proposal differs from both on the axis that matters for zero-shot transfer. We fit a distribution over cameras from cameras other than the deployment camera, sample from it during policy training, and never see the deployment sensor. We also test inside a closed control loop, where per-frame perception errors compound.
On the robotics side, Robust Visual Sim-to-Real Transfer for Robotic Manipulation supplies the method we build on. It provides realistic-texture randomization, lighting, color and camera-pose ranges, and an offline cube-localization proxy task for tuning ranges. It contains no sensor noise. It also constrains the design, since it reports that excessive randomization degrades policies. Our noise prior is therefore bounded by measured sensor statistics, not set arbitrarily wide. RoboTwin 2.0 likewise randomizes clutter, lighting and diffusion-generated backgrounds, but not the sensor. Encoder-Adapted Sim-to-Real Transfer of Simulation-Trained Diffusion Policies reports 0% direct zero-shot success and recovers by adapting the observation encoder on real demonstrations. It motivates the question of how much of that encoder gap is sensor appearance that could have been randomized away.
Research Program
The sensor model is applied in linear intensity space, before the camera's tone curve. Rendered sRGB frames are first mapped back to approximately linear intensity $x$. A sampled camera then produces
$$y = \mathrm{T}\Big( (1-M)\odot\big(g\,\mathrm{Poisson}(\phi\odot e\,x / g) + \epsilon\big) + M\odot s \Big),$$
where $e$ is exposure and $g$ is the gain, so that fewer photons per intensity level means stronger relative grain. $\phi$ is a per-pixel gain map with a randomized spread, representing fixed-pattern non-uniformity, which A convex 3D deconvolution algorithm for low photon count fluorescence imaging notes most models ignore. $\epsilon\sim\mathcal{N}(0,\sigma^2)$ is read noise. $M$ is a sparse binary mask with rate $\rho$ marking stuck pixels whose value $s$ is black or saturated. $\mathrm{T}$ is a randomized tone curve with white balance, followed by quantization. Each episode draws one camera $(e, g, \sigma, \rho, \text{spread of }\phi, \mathrm{T})$ and keeps the fixed pattern constant across the episode, as a physical sensor would. Shot and read noise are redrawn every frame.
The prior over these parameters is fitted from the raw noisy/clean pairs of the SIDD smartphones across their ISO settings. We estimate how read noise co-varies with gain per phone, then pool the phones into a joint prior with a modest margin. The width of that margin is itself a studied variable, because of the over-randomization result above. This whole step is a few elementwise operations per frame on the GPU. It should not dent the throughput of a simulator such as ManiSkill2, which reports 2532 frames per second for rendering. We will measure and report the overhead.
As a secondary, separable ingredient, we test a small auxiliary head on the policy encoder. It predicts a per-pixel reliability weight bounded in a fixed range, borrowing the design of the 3,211-parameter weight network in Learning Weighted Least Squares Data Term for Poisson Image Deconvolution. It is supervised in simulation by the known local noise variance and is used to reweight encoder features. This is the cheapest form of "noise-aware perception" we can think of that respects the throughput constraint. Iterative inversion solvers of the kind used in the infimal-convolution work are ruled out for the policy loop on cost grounds. The head is kept only if it adds measurable transfer beyond randomization alone.
Policies are behavior-cloned from simulated demonstrations, following the recipe of Robust Visual Sim-to-Real Transfer for Robotic Manipulation, on a Franka arm. The task set is a subset of tabletop pick, place and push tasks in that paper's style, plus ManiSkill2's PickCube, which has a reported sim-to-real reference.
Baselines
The naive reference is the strongest existing method with nothing added: the macro visual randomization recipe of Robust Visual Sim-to-Real Transfer for Robotic Manipulation, with realistic textures and lighting, color and camera-pose ranges tuned by the cube-localization proxy. That paper reports a 93% average real-world success rate across seven manipulation tasks. It is the strongest competitor because it already achieves high zero-shot success, and it embodies the reviewer's view that sensor effects are second order. For PickCube we additionally cite ManiSkill2, which reports 91% success in simulation and 60% on a physical robot zero-shot. Both are run by us on the same robot, cameras, demonstrations, architecture and training steps. Our variant changes only the observation corruption. Beating this baseline means matching it within noise under normal room lighting and exceeding it at the reduced illumination levels.
The second baseline is the practitioner's default: the same macro randomization plus additive Gaussian noise, Gaussian blur and brightness/hue jitter. Its ranges are tuned with the same proxy-task budget we give our own prior. The key ablation is a matched-variance Gaussian control. For each sampled camera we compute the per-pixel variance our model would produce and add zero-mean Gaussian noise of exactly that variance, independent of signal and with no fixed pattern or impulses. This isolates signal dependence and spatial structure from sheer noise magnitude. Further single-component ablations remove the fixed pattern, the impulses and the Poisson term in turn. A width ablation trains with the SIDD prior at its fitted width, half width and double width.
Two references bracket the result from above, and neither is held to the zero-shot constraint. One is a target-calibrated oracle in the spirit of Sensor Transfer, with parameters fitted to unlabeled frames from the deployment camera. The other is encoder adaptation on real demonstrations as in Encoder-Adapted Sim-to-Real Transfer of Simulation-Trained Diffusion Policies, which reports an average normalized task completion of 0.82 on MimicGen tasks versus 0% zero-shot. Closing a meaningful fraction of the gap to the calibrated oracle without target data is the success criterion against it. We do not expect to beat the oracle.
Why not just…?
Why not just add Gaussian noise and color jitter, which every pipeline already does? This is the real threat, and the matched-variance control exists to settle it. The prior evidence is suggestive but indirect. Learning Camera-Aware Noise Models shows that real camera noise is signal-dependent and structured, and that denoisers trained on realistic synthetic noise reach 48.71 dB on SIDD. That is a denoising result, not a control result. If matched-variance Gaussian noise yields the same low-light success curve as our model, we will report that the structure does not matter for policies and recommend the cheap augmentation.
Why not just run a good denoiser on the real camera stream at test time? A denoiser trained on camera-aware noise could make real frames look cleaner than any simulator's output. This is a fair baseline, so we include it: a DnCNN-style denoiser trained on SIDD-style synthetic noise, placed in front of the macro-randomization policy. Our expectation, which is untested, is that a denoiser trained on other cameras leaves residual artifacts and removes fine edges, the contrast-loss failure documented for variational denoisers in Infimal Convolution of Data Discrepancies for Mixed Noise Removal, and that these residuals are themselves out of distribution for a policy trained on clean renders. The added per-frame latency is also measured. If the denoiser front end matches us, the conclusion is that inversion at test time suffices and training-time randomization is unnecessary.
Why not translate simulated images to real-looking ones with an unpaired image translation network? Sensor Transfer reports that CycleGAN, UNIT and MUNIT translations distort geometry and hallucinate textures and are beaten by physical sensor models on detection. Such translation also needs target-domain images, which violates the zero-shot setting we study. We therefore do not rerun it, and we say so as a scoping decision.
Why not just collect ten real demonstrations and adapt the encoder? When demonstrations are available this works, per Encoder-Adapted Sim-to-Real Transfer of Simulation-Trained Diffusion Policies, and we report it as an upper reference. But that approach depends on tightly matched camera setups. The adaptation must also be repeated for every new camera and every lighting regime, which is exactly the variation our prior is meant to absorb.
Proposed Experiments and Decisive Checks
The first check is cheap and offline, so that the idea can be killed before any robot time. We hold out one SIDD phone at a time when fitting the prior. We train the policy encoder in simulation under each corruption scheme and measure two things on the held-out phone's noisy/clean pairs. One is the distance between the encoder's embeddings of the noisy and clean versions of each image. The other is cube-localization error on real low-light captures from the deployment cameras, reusing the proxy task of Robust Visual Sim-to-Real Transfer for Robotic Manipulation, which that paper shows correlates with manipulation success. If structured noise does not beat matched-variance Gaussian noise on the held-out phone here, we stop.
The main experiment is real-robot zero-shot evaluation. We use two RGB cameras absent from the prior: the Zed model used in DROID and a second depth-camera RGB stream. Workspace illumination is swept from normal room light down to dim levels where the camera's auto-gain rises sharply. Exposure and gain are left on automatic, as a deployed system would run. The primary metric is task success rate as a function of measured illuminance, with enough trials per cell to report binomial confidence intervals. The decisive pattern for the claim has three parts. All methods should be roughly tied at normal light. Macro-randomization and Gaussian-jitter policies should fall off as light drops. Our policy's curve should stay flatter, and the matched-variance control should sit with the Gaussian baselines rather than with ours. The ablations then attribute the effect to shot noise, fixed pattern or impulses. The width sweep tests whether the over-randomization penalty seen for macro parameters reappears for sensor parameters, which would show up as lost success under normal light. The auxiliary reliability head is retained only if it improves the low-light end of the curve beyond randomization alone. Finally, we report the training-throughput cost of the corruption step, to confirm it stays negligible relative to rendering.
References
- A convex 3D deconvolution algorithm for low photon count fluorescence imaging
- Image Reconstruction in Light-Sheet Microscopy: Spatially Varying Deconvolution and Mixed Noise
- Hybrid Model and Split Bregman Iteration Algorithm for Image Denoising
- Infimal Convolution of Data Discrepancies for Mixed Noise Removal
- Sensor Transfer: Learning Optimal Sensor Effect Image Augmentation for Sim-to-Real Domain Adaptation
- Learning Camera-Aware Noise Models
- Robust Visual Sim-to-Real Transfer for Robotic Manipulation
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- Encoder-Adapted Sim-to-Real Transfer of Simulation-Trained Diffusion Policies for Robot Manipulation
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- Learning Weighted Least Squares Data Term for Poisson Image Deconvolution
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset