Models trained on rendered images, where every pixel's pose and mask come for free, often fail on real camera images. A common fix is to pass each render through a neural "refiner," a network that edits a computer-graphics image so it looks like a photograph, and then train on the edited images. The problem is that a refiner trained only to look real can move edges, thicken silhouettes, or invent texture. When that happens, the free labels stop matching the pixels, and the downstream model learns from slightly wrong supervision. We propose to constrain what the refiner is allowed to change. Object appearance is corrected only through a learned residual texture that lives on the 3D surface and is rendered by the same rasterizer that produced the labels, so it cannot shift an object's outline. Camera effects are corrected only through a small, physically parameterized sensor model. Unconstrained pixel editing is allowed only inside a thin band around silhouettes and contact regions, where compositing errors actually occur.
The claim is that label preservation by construction, not photorealism, is the property that decides whether refined synthetic data closes the sim-to-real gap for geometric tasks. The effect should be largest under occlusion, where small silhouette errors matter most. We test this on 6D object pose estimation (recovering an object's 3D rotation and translation from an image) on LINEMOD, Occluded-LINEMOD and YCB-Video. These benchmarks have a property we exploit for diagnosis: every real test image comes with a ground-truth pose and a CAD model, so we can render an exactly aligned synthetic twin and measure how far any refiner drifts from real geometry.
What exists and what is missing
Post-rendering refinement is well developed as a graphics tool. Lifting AutoEncoders and MOST-GAN add adversarial image-space stages on top of coarse 3D face renders. GRMM uses a lightweight image-space network to sharpen Gaussian-splat renders. From NeRFLiX to NeRFLiX++ restores NeRF outputs using a learned degradation simulator and reports PSNR gains of 0.6 to 1.3 dB. All of these optimize visual fidelity. None asks whether the refined pixels still agree with an annotation. We reuse their basic structure, a coarse render followed by a corrective network, but change both what the network may edit and what it is judged on.
Three papers already join refinement to sim-to-real training, so the obvious combination is taken. Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (PixelDA) translates synthetic images toward real ones in image space, using a masked content loss to discourage foreground hallucination. Its authors restrict it to low-level appearance shifts. Neural Object Learning for 6D Pose Estimation Using a Few Cluttered Images (NOL) synthesizes pose training images with a super-resolution decoder and documents background pixels bleeding into object silhouettes. HOGSA sharpens Gaussian-splat hand renders with an image-space network before training a hand tracker.
All three let an image-space network touch every pixel, and they rely on a soft loss to keep geometry in place. Our proposal differs in one structural respect: the refiner's ability to move labeled geometry is removed architecturally rather than discouraged by a loss. We also add a direct measurement of geometric drift, which none of them report.
The constraining evidence cuts both ways. Learning to Estimate Object Poses without Real Image Annotations crops away synthetic backgrounds because background pixels hurt transfer. That suggests appearance near and around the object matters more than global realism. Segmentation-Driven 6D Object Pose Estimation shows that bounding-box-level reasoning breaks under occlusion, the regime where accurate silhouettes are most valuable. ReSim composites splatted backgrounds with ray-traced CAD foregrounds using a depth buffer and leaves the seam between them unrefined. That is exactly where our band-limited correction is aimed.
Core Claim
For geometric downstream tasks, a refiner should be judged by two quantities measured separately: how much it shrinks the appearance gap, and how much it moves geometry. Unconstrained image-space refiners trade the second for the first. A refiner whose object-appearance edits are attached to the 3D surface can reduce the appearance gap comparably while keeping geometric drift near zero. That difference should show up as better pose accuracy, especially on Occluded-LINEMOD.
Research Program
The refiner has three parts.
The first part is a surface residual. This is a small network over surface position, viewing direction and a random lighting code. It outputs a color correction for every point on the CAD model, and the differentiable rasterizer renders that correction into the image. The object's pixel footprint is fixed by geometry, so this part cannot change the mask, keypoints or edges by construction. It can still add specular highlights, scan-texture correction and shading variation.
The second part is a sensor model with a few dozen parameters: blur kernel, noise level as a function of intensity, a color matrix, gamma and vignetting. It is applied to the whole image. Its low capacity means it can imitate the camera but cannot invent structure.
The third part is a boundary network. This image-space network may edit pixels only within a few pixels of rendered silhouettes and object-to-table contact lines. It handles soft edges, shadows and blending against real or splatted backgrounds, the error NOL reports and ReSim leaves open.
All three parts are trained jointly against an adversarial critic on unlabeled real images. Real object patches are located by a detector trained on raw synthetic data. No real pose labels are used for training.
On the analysis side, we use the ground-truth poses in LINEMOD-family test sets only for evaluation. For each real image we render its aligned synthetic twin, refine it with each method, and compute two scores. The first is a geometric drift score: the mean distance between edges in the refined image and edges in the real image along the true silhouette, plus the change in mask area that a fixed segmenter detects. The second is an appearance gap score: the feature distance between the refined patch and the real patch inside the true mask. This yields a drift-versus-gap plot for every refiner and lets us test whether drift, rather than residual gap, predicts downstream pose error.
Baselines
The naive reference point is training the pose network on raw synthetic renders, with the standard randomization of lighting and backgrounds and no refinement. The downstream architecture is fixed throughout to PVNet with bounding-box cropping, following Learning to Estimate Object Poses without Real Image Annotations. Every method uses the same renders, the same number of training images and the same pose-network schedule.
Pure domain randomization is the second baseline, following the recipe of Indirect Object-to-Robot Pose Estimation from an External Monocular RGB Camera, which reports about 2 cm accuracy and 91% grasp success on a robot. We give it a rendering budget matched in GPU-hours to our refiner's training plus generation cost, so that the extra realism we buy is compared against simply rendering more varied data.
The strongest unconstrained refiner is PixelDA. It reports a mean angle error of 23.5° on synthetic-to-real LineMod, against 56.58° for DANN and 53.27° for DSN, in Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks. We re-implement it with the same critic, the same unlabeled real images and the same hyperparameter search budget as ours. We also run an off-the-shelf NeRFLiX++-style restorer to represent fidelity-optimized refinement.
The strongest existing data-synthesis method in the file is NOL, which reports a BOP score of 37.7 (RGB only) on Occluded-LINEMOD using about 14 real images per object, against 14 to 20 for other synthetic-only methods. Because NOL uses a few posed real images while we use none, we report it as a reference ceiling rather than a matched competitor. We also run a variant of our method that is given the same 14 images for a fair comparison.
The ablation that isolates the key ingredient keeps our critic, data and parameter count, but replaces the surface residual and band-limited boundary network with a single full-image residual network. Beating it shows that the architectural constraint, not the training signal, is responsible for any gain.
Beating a baseline means a statistically reliable improvement over three seeds in ADD on LINEMOD, BOP score on Occluded-LINEMOD, and recall on YCB-Video. On Occluded-LINEMOD we expect a larger margin than on LINEMOD.
Why not just…?
Why not just randomize the renderer harder? Domain randomization is cheap and strong on textured objects. The authors of Indirect Object-to-Robot Pose Estimation themselves note that it degrades on texture-less and pseudo-symmetric objects, and LINEMOD contains several of each. Randomization widens the synthetic distribution but does not move it toward the real one. Our compute-matched comparison settles the question. If randomization equals us on the texture-less LINEMOD objects, the surface residual is not earning its cost, and we would report that.
Why not just use real depth and ICP self-training? This is the most serious threat. Learning to Estimate Object Poses without Real Image Annotations reaches 88.4% ADD on LINEMOD and raises YCB-Video recall from 48.6% to 77.1% by refining pseudo-labels with depth (ICP aligns the CAD model to the depth point cloud), close to the roughly 91.4% of fully supervised training. Our method cannot be expected to beat that in the RGB-D setting. We therefore position it in two ways. In the RGB-only setting, where depth is unavailable or unreliable on shiny and transparent surfaces, it is a standalone method. In the RGB-D setting, it is a better initializer for that same self-training loop. The decisive test is whether starting their pipeline from our refined-data model rather than raw synthetic data raises final accuracy or reduces the number of rejected pseudo-labels. If it does neither, the refiner adds nothing once depth is present.
Why not just add a stronger content loss to an image-space refiner? A reviewer could argue that PixelDA plus an edge-alignment penalty would match us without 3D machinery. The full-image ablation, run with an explicit silhouette-consistency loss, tests this directly. The drift diagnostic shows whether soft penalties keep edges in place as tightly as the architectural constraint does. If the penalized image-space refiner reaches the same drift and the same pose accuracy, the simpler design wins and the contribution reduces to the diagnostic.
Why not just align features instead of pixels? The evidence in PixelDA already answers most of this. Feature-level adaptation more than doubled the angle error relative to pixel-level adaptation on LineMod, and pixel-level outputs can train any architecture without redoing the adaptation. We do not rerun feature-level methods beyond one DANN check.
Proposed Experiments
The first decisive check is the drift study on LINEMOD and Occluded-LINEMOD aligned twins. We predict that every unconstrained refiner shows silhouette edge drift that grows with refiner capacity, that ours stays near the raw-render level, and that both reach comparable appearance-gap reductions. If ours does not reduce the appearance gap appreciably, the constraint is too strong and the method fails at its first step.
The second check relates drift to downstream error. Across all refiners and capacities, we regress ADD on Occluded-LINEMOD against drift and against appearance gap. The core claim predicts that drift explains more of the variance than gap, and that the slope is steeper on Occluded-LINEMOD than on LINEMOD, consistent with the roughly 30% occlusion penalty that BOP: Benchmark for 6D Object Pose Estimation documents. The opposite result would mean realism matters more than geometry and would overturn the claim.
The third check is the main table: ADD on LINEMOD, BOP score on Occluded-LINEMOD, and recall on YCB-Video for all baselines, in RGB-only and RGB-D self-training settings. Reference points are NOL's 37.7 BOP score on Occluded-LINEMOD and the 88.4% ADD and 77.1% YCB-Video recall from depth self-training.
A component ablation removes the surface residual, the sensor model and the boundary network one at a time. We expect the boundary network to matter most on YCB-Video clutter and the surface residual to matter most on reflective objects.
If time allows, we apply the same constraint to HOGSA's hand renders on the Arctic benchmark, against its reported mean per-joint error of 20.96 mm, to test whether the principle carries over to articulated, non-rigid geometry.
References
- Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion
- MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation
- GRMM: Real-Time High-Fidelity Gaussian Morphable Head Model with Learned Residuals
- From NeRFLiX to NeRFLiX++: A General NeRF-Agnostic Restorer Paradigm
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Neural Object Learning for 6D Pose Estimation Using a Few Cluttered Images
- HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation
- Learning to Estimate Object Poses without Real Image Annotations
- Segmentation-Driven 6D Object Pose Estimation
- Re$^3$Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation
- Indirect Object-to-Robot Pose Estimation from an External Monocular RGB Camera
- BOP: Benchmark for 6D Object Pose Estimation