Detectors trained on images from game engines lose much of their accuracy on real camera footage. One reason gets little attention: a game engine and a real camera turn light into pixels very differently. A real camera filters light through color filters with their own spectral sensitivities. It fills each pixel's finite charge well until the pixel saturates, amplifies the signal with gain settings that do not trade off cleanly against exposure time, adds noise that depends on the signal, and then runs a proprietary pipeline of non-linear tone and color curves. A game engine applies one fixed, idealized tone curve to its internal brightness values and saves an 8-bit image. The best-known fix, Modeling Camera Effects to Improve Visual Learning from Synthetic Data, adds camera effects to these finished 8-bit images. That is too late in the pipeline. By then the engine's own curve has already decided which highlights are clipped and how shadows are compressed, so the augmentation can only distort an image that was already saturated, never re-saturate the underlying light.
We propose to capture the engine's scene-linear, high-dynamic-range brightness buffer before any tone mapping, and to pass it through a randomized but physically parameterized camera. That camera samples spectral sensitivities from a measured low-dimensional space, samples an exposure and gain setting with their measured non-linear interaction, clips at a sampled well depth, adds Poisson-Gaussian noise, and applies a sampled non-linear and possibly spatially varying tone curve. In two sentences: train on synthetic images where the camera, not the engine, decides what saturates and how brightness is compressed, and randomize that camera across the space of real sensors. If the claim holds, the gains should appear exactly where the post-hoc approach cannot reach, namely in high-dynamic-range regions such as sun glare, tunnel exits and headlights at night.
Core Claim
Randomizing the camera model on linear high-dynamic-range renders closes more of the sim-to-real gap for detection than applying the same randomization to tone-mapped 8-bit renders. The advantage should concentrate on image regions where scene brightness exceeds what an 8-bit tone-mapped image retains. The ingredient being tested is where the camera model sits in the pipeline, not the camera model itself. For that reason the central comparison holds the camera model fixed and varies only its input: the true linear buffer, versus an 8-bit render converted back to approximately linear values by inverting its tone curve.
The prior work plays three different roles here.
Some of it motivates the claim.
- Modeling Camera Effects shows that sensor effects matter for detection, reporting a 4.27 AP gain for cars over its 21K-image baseline when training on Virtual KITTI and testing on KITTI. Its pipeline operates post-render with CIELAB color shifts, a perceptual color space that separates lightness from color.
- Virtual KITTI 2 shows that better physically based materials and lighting roughly halved the gap. That suggests photometry matters but is not exhausted by better shading.
Some of it supplies the method.
- What is the space of spectral sensitivity functions for digital color cameras? measures 28 cameras and finds that two principal components explain over 97% of the variance in their spectral sensitivities. This gives a compact, realistic space from which to sample.
- Exploring multispectral reconstruction based on camera response prediction provides a forward imaging model with an explicit correction for the non-linear exposure–gain interaction. We reuse its structure for the exposure stage.
- Blind Image Deblurring with Outlier Handling shows that clipped pixels behave as structured outliers, not Gaussian noise: linear solvers reach an error ratio of 18.04 against 3.09 with explicit outlier handling. This supports modeling saturation as a hard physical event.
Some of it constrains the design.
- The same multispectral paper finds that simulated characterization still trails real capture (5.73% versus 2.63% RMSE). So we treat the camera model as a randomization prior, not a calibrated replica of one camera.
- RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images shows that pretrained backbones are sensitive to the non-linear statistics of their inputs. This warns that any ISP choice at training time can become a shortcut the network exploits.
This differs from the closest work in specific ways. Modeling Camera Effects never sees light brighter than the engine's white point and does not vary spectral sensitivity. Every Camera Effect, Every Time, All at Once models lens geometry and rolling shutter but stops before radiometry. DiET-GS learns a camera response function for one scene's reconstruction using an event camera, rather than randomizing over response functions for semantic training.
Research Program
Synthetic source. We use CARLA. It is open source, which lets us disable the Unreal tone mapper and write floating-point scene-linear render targets alongside the standard tone-mapped output and the detection labels. Every scene is rendered once and stored in both forms, so all conditions share identical geometry, labels and sample counts. The weather and lighting presets give natural variety in dynamic range, including low sun, night with headlights and tunnels. We will also try re-rendering Virtual KITTI 2. If linear buffers prove unobtainable there, we report that and stay with CARLA.
The camera stage applies these steps in physical order:
- Spectral response. Engines render RGB, not full spectra. We approximate each sampled camera by a 3×3 matrix. To build it, we draw a spectral sensitivity from the two-component space of the 28-camera database. We then fit the least-squares map from sRGB-primary responses to that camera's responses over a standard set of reflectances and illuminants. This is the weakest physical link, so it gets its own ablation.
- Exposure and gain. We sample an exposure time and gain and apply the non-linear exposure–gain correction in the form used by the multispectral virtual-characterization model.
- Saturation. We clip at a sampled full-well level in linear units, so that sun-lit surfaces and headlights saturate the way they would in hardware.
- Noise and quantization. We add signal-dependent Poisson-Gaussian noise and quantize as an analog-to-digital converter would, at 10–14 bits.
- Rendering to the final image. We apply a sampled white balance, a sampled global non-linear tone curve, and, with some probability, a local tone-mapping operator that varies over the image, as phone and automotive ISPs do.
All parameter ranges come from published measurements where the file supplies them (spectral sensitivities, exposure–gain behavior). Otherwise they are set wide and then reported.
Analysis. Beyond overall accuracy, we label each real test object by how much of its bounding box falls in near-saturated or crushed-shadow pixels. The claim predicts that the advantage of the linear-input variant grows with that fraction. We also train the same model on real RAW data, testing on PASCAL RAW, which ends in linear sensor counts rather than sRGB. This checks whether stopping the camera stage before the tone curve transfers better to a linear-input detector.
Baselines
Do-nothing reference. A Faster R-CNN trained on CARLA's default tone-mapped output with only flips and crops, the same detector family as Driving in the Matrix. For calibration on a different engine, that paper reports 0.6803 mAP on KITTI Easy for GTA V Sim 200k and 0.6247 for a Cityscapes-trained model. No reported result was found on Lacuna for CARLA-trained detectors evaluated on KITTI, so this baseline must be measured here.
Strongest existing method. The post-render sensor pipeline of Modeling Camera Effects: chromatic aberration, blur, an exposure model with clipping, Poisson-Gaussian noise, and CIELAB color shift. It is applied to the same CARLA tone-mapped images with its published parameter ranges. Its reported 4.27 AP gain for cars over the unaugmented 21K baseline on Virtual KITTI to KITTI is the effect size to beat. We will also reproduce it on Virtual KITTI to check that our implementation recovers a gain of similar size. Every arm uses the same images, backbone, schedule, number of augmented samples per epoch, and a matched hyperparameter search of equal trials over each method's own randomization ranges.
Key ablation. Our full camera model applied to tone-mapped renders that have been converted back to approximately linear values by inverting the engine's tone curve, an "unprocessing" step. This isolates the one thing only linear buffers provide: brightness beyond the engine's white point. Further ablations remove one stage at a time: the spectral matrix, the exposure–gain non-linearity, the local tone mapping, and the saturation stage.
Ceiling reference. Fine-tuning on a few hundred labeled real KITTI images. It indicates how much of the gap is left for any synthetic-only method to close.
Beating the strongest existing method means two things. The full linear method must gain at least the reproduced post-hoc gain plus a margin outside the seed-to-seed spread over five seeds on KITTI car AP. And the advantage must be larger on objects in the high-saturation group than on the rest.
Why not just…?
Why not just use color jitter and Gaussian noise? Modeling Camera Effects reports that its physical pipeline clearly outperformed PCA color shift, elastic deformation and additive Gaussian noise. The deblurring results show Gaussian treatments of clipping failing badly. We still include a tuned color-jitter arm, because the earlier comparison used a different engine.
Why not just apply the post-hoc pipeline to inverted sRGB, avoiding engine access? This is the real threat. If inverting the tone curve recovers most of the useful signal, the linear-buffer requirement is unnecessary, and a much simpler tool applicable to any existing dataset would suffice. The key ablation tests this directly. If the inverted-sRGB arm matches the linear arm, including on high-saturation objects, we will report that the benefit comes from the camera model and not its placement, and recommend the simpler pipeline.
Why not just render more data or use a more photorealistic engine? Driving in the Matrix shows continued gains up to 200k images but a persistent gap. Virtual KITTI 2 halved the gap without eliminating it, and degraded under lighting and weather changes. We include a data-scaling arm with 4× unaugmented images to test whether volume alone substitutes for the camera stage.
Why not just adapt at test time with a learnable ISP? A RAW-Adapter-style input adapter needs target-domain training and labels. Our setting is synthetic-only. The two approaches can be combined, and the PASCAL RAW experiment asks whether our training makes such adaptation less necessary.
Proposed Experiments
The primary experiment trains on CARLA and tests on KITTI 2D car and pedestrian detection and on Cityscapes detection, reporting AP with five seeds per arm, together with the saturation-stratified breakdown. The first decisive check compares linear-input against inverted-sRGB-input with the camera model held fixed, on the high-saturation group. A gap there, absent on well-exposed objects, supports the claim. No gap anywhere refutes it.
The second check tests spectral variation without stopping at the training camera. We train with spectral sensitivities sampled from half of the 28 cameras and test robustness on real images color-transformed to the held-out cameras' responses. The multispectral reconstruction study shows that a change of illuminant alone can raise RMSE from 2.87% to 41.32%. If the spectral matrix ablation costs nothing here, we drop that component.
The third check trains on CARLA and tests on PASCAL RAW in dark conditions, with the camera stage truncated before the tone curve. This is compared against the same detector trained on tone-mapped synthetic data and then re-linearized. RAW-Adapter's real-trained 82.5 mAP serves as an upper reference, not a target.
References
- Modeling Camera Effects to Improve Visual Learning from Synthetic Data
- Virtual KITTI 2
- What is the space of spectral sensitivity functions for digital color cameras?
- Exploring multispectral reconstruction based on camera response prediction
- Blind Image Deblurring with Outlier Handling
- RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images
- Every Camera Effect, Every Time, All at Once: 4D Gaussian Ray Tracing for Physics-based Camera Effect Data Generation
- DiET-GS: Diffusion Prior and Event Stream-Assisted Motion Deblurring 3D Gaussian Splatting
- CARLA: An Open Urban Driving Simulator
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?