Real Errors, Synthetic Answers: Mining Part Confusions on Real Images to Steer What the Simulator Renders

Teaching a model to segment the parts of an object, such as the doors, wheels, headlights and mirrors of a car, normally requires people to paint every part in thousands of photos. A cheaper route is to render 3D computer-aided design (CAD) models, which give perfect part masks for free, and then transfer to real photos. The transfer is where things break. On the synthetic-vehicle benchmark of Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles, a model trained only on renders reaches 37.19 mean intersection-over-union (mIoU, the average overlap between predicted and true part masks) on real cars. The paper's best adaptation method reaches 49.93, and failures concentrate on small parts.

This note borrows the oldest trick in detection, iterative hard negative mining: find the examples the model gets confidently wrong, add them to training, and repeat. The proposal changes where each half of that loop runs. Errors are found on unlabeled real photos, using label-free signals, because only the real domain shows the failures that matter. Corrections are produced in the simulator, by rendering new scenes chosen to reproduce each specific confusion. The renders carry exact labels, so the model never trains on a guessed label for the hard cases. In two sentences: the real photos say which part confusions to fix, and the renderer supplies perfectly labeled examples of exactly those confusions. This should beat both rendering more random data and self-training on noisy real pseudo-labels, because it targets the real failure modes without inheriting label noise.

Core Claim

The claim is that, at a fixed rendering budget, real-domain error signals can steer simulator sampling and improve synthetic-to-real part segmentation more than two alternatives: random extra renders, and hard examples mined on synthetic data alone. The gains should concentrate on the specific part-pair confusions that were targeted.

Three observations from the file motivate this. First, the research file's summary of hard negative mining notes that negatives mined on synthetic images often fail to represent failures on physical sensors. Mining therefore has to see real data. Second, the same summary notes that mining on unlabeled pools risks "false hard negatives," where a correct prediction is punished because the pool label is wrong. Our design avoids this because real images only select what to render and never supply training labels for the hard cases. Third, Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles and its dataset companion CGPart already give us the tools to connect a real image to the simulator.

That method, called GMG, has three steps. It finds the nearest CAD model and viewpoint for a real image. It warps the synthetic part masks onto the real vehicle with a thin-plate spline, a smooth 2D deformation. It then keeps only the warped labels that pass a confidence filter, and trains on them. GMG supplies our method's plumbing, but it runs once, uses a static pool of renders, and never asks the renderer for anything new.

The closest mining-in-simulation work is RLS3. It trains a reinforcement-learning agent to rearrange Unity scenes until a vision-language model fails. It differs from our proposal in three ways: its failure signal comes from inside the simulator, it treats objects as rigid boxes, and its authors state it was never tested on real camera feeds. We take its core idea, that the simulator should be driven toward the model's decision boundary, and drive it from real-image failures at the level of parts instead.

Research Program

The loop has four stages, repeated for several rounds.

Stage 1: train and predict. We train a segmentation model on the current synthetic pool plus GMG pseudo-labels, then predict parts on the unlabeled real training images.

Stage 2: detect errors without labels. We use three signals, each chosen because it does not depend on ground truth.

The first is disagreement with confident warps. Where GMG's warped CAD masks are reliable but the model predicts a different part, we record a confusion pair, such as front door predicted as back door.

The second is geometric implausibility. Given the CAD model and viewpoint that GMG matched to the image, each part has an expected region and expected neighbors, so a wheel predicted above the window line counts as an error. Left/right consistency adds one more check: horizontally flipping the image should swap left and right part labels, and a violation flags a confusion.

The third is background false positives. These are confident part predictions far outside the vehicle silhouette implied by the warp. This is classic bootstrapping in the style of Human Face Detection in Visual Scenes, which found that appending misclassified background patches mattered more than adding random background. Patches far from the object are safe to treat as background, so we crop them.

Stage 3: turn errors into render requests. Each mined error becomes a specification for new synthetic images. The renderer uses the matched CAD model family and samples viewpoints near the matched one, since part confusions tend to cluster at particular viewing angles. It varies lighting and materials on the two confused parts. It composites the mined real background patches behind the rendered vehicle, where they are labeled background exactly. The requests are weighted by how often each confusion pair occurs, so frequent real failures get more of the budget.

Stage 4: render, translate, retrain. New renders are optionally passed through a pixel-level style translator in the manner of PixelDA. PixelDA uses a generative adversarial network, a generator trained against a discriminator, plus a foreground content loss so that labels survive the translation. The renders are then added to the pool and the model is retrained.

The scientific questions are which of the three error signals carries the gain, whether the loop overfits to simulator artifacts, and whether gains appear on the targeted confusions. The overfitting concern comes from the file's warning, illustrated by jet-tagging networks that learned Monte Carlo artifacts (Jet substructure classification).

Baselines

The naive reference is the source-only model trained on the original synthetic UDA-Part renders. The file reports 37.19 mIoU on the car category for it (Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles). We will retrain it with our backbone so that every comparison shares the same architecture.

The strongest existing method is GMG from the same paper, at 49.93 mIoU on cars. The same paper reports 53.77 when ground-truth viewpoints are given. That figure serves as an alignment ceiling: if our loop exceeds it without true viewpoints, the gain cannot be explained by better pose matching alone. For the other four vehicle classes and for PascalPart, I found no reported result on Lacuna in the file, so we will report rerun numbers only. GMG is the strongest competitor because it already exploits the CAD-to-real link that our method depends on. Any gain we show must come from the closed loop, not from the warp. We will rerun GMG with our backbone, our training schedule and our hyperparameter search budget, and require a clear improvement over the rerun, not over the published number.

The baseline that isolates our key ingredient is budget-matched random rendering. It is GMG plus the same number of extra renders as our loop adds, sampled uniformly over CAD models, viewpoints and lighting. A second baseline, synthetic-side mining, uses the same budget but chooses renders from errors measured on a held-out synthetic set. This is online hard example mining run entirely inside the simulator. A third baseline is iterated GMG self-training: the same number of rounds, but each round re-pseudo-labels the real images with the current model and does no new rendering. Our method beats these baselines only if it exceeds all three by more than the spread across seeds at equal compute. Within the method, we ablate each of the three error signals separately.

Why not just…?

Why not just render more random synthetic data? This is the most serious threat. Rendering is cheap, and domain randomization might cover the hard viewpoints by chance. We will run random rendering at one, two and four times our budget. If four-times random matches our loop, the honest conclusion is that mining buys efficiency, not accuracy, and the note will report it that way. The evidence for expecting more is indirect. Hard Negative Contrastive Learning for Fine-Grained Geometric Understanding found that 4K targeted image negatives helped more than 100K retrieval negatives. RLS3 reported roughly 40% better sampling efficiency than random scene generation.

Why not just iterate self-training on real pseudo-labels? Self-training uses real pixels directly, which our renders never fully match. However, its errors compound on exactly the hard cases, because a confused part produces a confused pseudo-label. The Over-Certainty Phenomenon in Modern UDA Algorithms shows that adaptation through confident self-labels drives models toward overconfident mistakes. The iterated-GMG baseline tests this directly. We will also measure calibration error on the targeted parts.

Why not just generate hard negatives with diffusion, as GeNIe does? GeNIe partially noises an image and then denoises it under a competitor class's text prompt, which produces strong classification hard negatives. For segmentation, though, the edited pixels have no trustworthy part mask. Diffusion edits also cannot change viewpoint, and viewpoint is where part confusions live. If time permits, we will include a GeNIe-style variant that edits rendered images and keeps only edits that preserve the mask. This will show whether 2D editing recovers part of our gain.

Why not just style-translate the renders? Translation in the manner of PixelDA fixes texture and noise. Its own authors state that it cannot repair structural mismatches. We include translation inside the loop and also as a stand-alone baseline. Our loop should add gain on top of translation, not replace it.

Proposed Experiments and Decisive Checks

The main experiments use UDA-Part/CGPart. Training uses the 126K synthetic renders from 21 CAD models across five vehicle classes. Evaluation uses the 200 annotated real PASCAL3D+ images. We report mIoU and per-part IoU, with small parts reported separately. PascalPart serves as an out-of-benchmark generalization test.

Three checks decide whether the claim holds.

The first is the budget-matched comparison: real-steered, synthetic-steered and random rendering, each run across several seeds.

The second is a causal targeting test. In one round we target only a random half of the mined confusion pairs. The claim predicts that real-image confusion rates fall significantly more for the targeted pairs than for the untargeted ones. If both halves improve equally, the gain is generic extra data, not mining.

The third is an artifact check. A probe classifier is trained to tell renders from real images using the segmentation features, and we track its accuracy across rounds. If the probe gets steadily easier while real mIoU plateaus, the loop is learning simulator shortcuts. In that case, the background-compositing and translation stages are the levers to test.

A negative outcome on the first check, with a positive outcome on the second, would still be informative. It would mean real errors identify the right fixes but the renderer cannot supply them faithfully enough.