Video-game engines can label every pixel of a street scene for free, but a segmentation model trained on those scenes loses much of its accuracy on real city streets. Rare, safety-relevant objects such as buses, trains, motorcycles and traffic lights suffer most. One natural remedy is to take a game's label map, optionally paste in extra rare objects, and ask a diffusion model (a generator that turns random noise into an image by removing noise step by step) to paint a photorealistic scene that follows the map. The catch is that these generators do not reliably obey the map. They drop small objects, invent ones that are not there, and blur boundaries, so a pixel labelled "bus" may show road. Filtering out such broken samples is well studied for text and tables, but the usual filters discard anything the model finds hard. In a long-tailed problem, "hard" mostly means "rare", so these filters would throw away exactly the examples augmentation was supposed to add.
This note proposes a region-level filter with a class-relative reference. We track how confidently the segmentation model predicts each generated object region over the course of training, then compare that region's trajectory with the trajectories of pixels of the same class in the original simulator images, whose labels are correct by construction. A generated "train" region is removed only if it looks worse than genuine simulator trains look to the same model at the same point in training, not because trains are hard in general. The claim is that this single change, a same-class reference taken from label-correct simulator data, makes generative augmentation reliably help rare classes in synthetic-to-real segmentation. Unfiltered augmentation and class-blind filtering do not.
Core Claim
Generative augmentation for the sim-to-real gap fails in two ways, and each pulls against the other. Generation errors are spatially local: a scene can be mostly good with one hallucinated or missing object. Rare-class examples are also intrinsically low-confidence. Image-level filtering wastes the good pixels. Class-agnostic confidence filtering, in the style of Curated LLM, which removes LLM-generated tabular rows using an auxiliary classifier's confidence and uncertainty over training, would preferentially delete rare classes. We claim that per-region, per-class filtering calibrated against simulator pixels of the same class removes label-image mismatches while keeping hard but correct rare examples. If so, the gains from generated data should concentrate in the rare classes that DAFormer and FREDOM identify as the bottleneck.
Concretely, let $\bar p_r$ be the model's mean predicted probability for the assigned label, averaged over pixels of generated region $r$ and over $K$ training checkpoints. Let $F_c$ be the empirical distribution of the same quantity for pixels of class $c$ in the original simulator images at the same checkpoints. A region of class $c$ is kept for training if
$$F_c(\bar p_r) \ge \tau,$$
where $\tau$ is a single percentile threshold shared across classes. Rejected regions are set to "ignore" in the loss; they are not deleted along with the rest of the image. A variability term, the standard deviation of confidence across checkpoints, is handled the same way. The reference $F_c$ moves with training, so the filter asks how an object compares with correctly labelled objects of its own kind, never how easy it is.
Where This Sits
Three pieces of prior work motivate the proposal. SimGen renders simulator conditions into realistic driving images with a two-stage diffusion cascade and reports a +4.4 mIoU downstream gain for a bird's-eye-view perception model. It uses the generated data without any per-sample quality control. Driving with DINO names the problem we target, the "consistency-realism dilemma". High-level conditions such as semantic layouts yield realism but hallucinate or omit objects. It attacks the problem on the generator side by conditioning on pruned DINOv3 features, and it does not examine class imbalance. We accept that any layout-conditioned generator will make some of these mistakes, and we remove them on the learner side. Adaptation of domain-specific transformer models with text oversampling supplies a warning: naive generative oversampling lowered CT-BERT accuracy from 77.70% to 76.14%. Unfiltered augmentation is therefore not a safe default.
Two works supply method. Curated LLM supplies the idea of filtering by learning dynamics, but it works on whole tabular rows, uses a class-blind criterion, and has no label-correct reference set. We add spatial granularity and the simulator-anchored, class-conditional reference. Emancipating Event Extraction filters LLM-recombined event schemas with an LLM "rationality check"; its idea of recombining gold fragments into new samples is the text analogue of our rare-object pasting.
Two further works constrain the design. Domain Adaptation for Rare Classes Augmented with Synthetic Samples shows that restricting adaptation to the synthetic-augmented rare class (DeerDANN, +24.0%) beats adapting all classes at once. It also reports that augmenting a rare class slightly hurt common classes. Our filter is likewise class-specific, but it acts on the data rather than on a feature-alignment loss, and we must check for the same common-class cost. DAFormer's Rare Class Sampling oversamples existing simulator images that contain rare classes and synthesizes nothing new. FREDOM rebalances through a fairness loss and structural context. Both are complementary to us, and both are the references we must beat.
Research Program
The pipeline has three stages, built on the public lhoyer/DAFormer codebase.
In the first stage we build a generated pool. We use a layout-conditioned diffusion image generator trained only on real images, with no Cityscapes labels. It renders GTA5 and SYNTHIA label maps in two modes. In the first mode it renders the original maps. In the second it renders edited maps into which rare-class instance masks (bus, train, truck, motorcycle, bicycle, rider, traffic light, traffic sign) are copied from other simulator frames at plausible positions, so that the rare classes gain new layouts and contexts.
In the second stage we collect learning dynamics online. During the first phase of ordinary DAFormer training on simulator data plus the generated pool, we log, at $K$ checkpoints, per-region confidence on the generated images and per-class confidence histograms on the simulator images. This costs only forward passes on data the model already sees.
In the third stage we apply the class-relative rule above, mark rejected regions as ignored, and continue training for the remaining iterations.
The analysis side matters as much as the accuracy numbers. We will measure the filter's precision and recall directly. To do this we audit a random sample of generated regions by hand, labelling each as matching or not matching its label, and we inject known corruptions, such as swapping a region's label or erasing an object from the conditioning map after generation. The central diagnostic compares, class by class, the fraction of correct generated regions retained under class-blind filtering versus class-relative filtering. The hypothesis predicts that class-blind filtering discards correct rare-class regions at a much higher rate, and that class-relative filtering does not.
Baselines
The do-nothing reference is DAFormer with Rare Class Sampling trained on GTA5 alone, adapting to unlabelled Cityscapes with its standard self-training. The reported figure is 68.3 mIoU on GTA5→Cityscapes, including a rise in train IoU from 16.0 to 65.1 (DAFormer). We rerun it with three seeds in our environment rather than quoting it, because the comparison must share hardware, iteration count and augmentation settings.
The strongest reported method on this benchmark in the file is FREDOM, which reports 73.6% mIoU on GTA5→Cityscapes (FREDOM). Its gain comes from loss-level rebalancing, which targets the same rare-class weakness. We will reimplement its fairness class-balance loss inside the DAFormer code and run it with the same budget. Our filtered augmentation is data-side and composes with it, so we also report FREDOM-loss plus our data. If our reimplementation falls short of 73.6, we report both numbers and say so.
Unfiltered generative augmentation is the SimGen-style use of the full generated pool with no curation, at the same mixing ratio. It isolates whether curation matters at all.
Image-level learning-dynamics filtering, which drops whole generated images with low mean confidence as a direct transfer of Curated LLM, isolates the value of region granularity.
Region-level class-blind filtering uses the same per-region statistics but a single global confidence threshold instead of the class-relative reference. This is the key ablation: it differs from our method only in the reference distribution.
For every generated-data condition we match total training iterations, the real-to-generated sampling ratio, and the size of the generated pool, and we tune each method's threshold on the same held-out split of simulator data, never on Cityscapes. Our method counts as a win only if both of the following hold across three seeds. First, it beats region-level class-blind filtering and unfiltered augmentation in mean IoU over the rare classes by more than the seed spread. Second, it does not lower overall mIoU or common-class IoU relative to DAFormer with Rare Class Sampling. Beating the rerun FREDOM loss is the stronger bar, reached either alone or by stacking.
Why not just…?
Why not just use Rare Class Sampling or a fairness loss and skip generation? These methods reweight the rare examples the simulator already contains. They cannot add new layouts, backgrounds or appearances for a class like "train" that occurs in only a handful of GTA5 frames. That is the variety the generated pool supplies. Evidence that generated data adds something beyond the backbone and sampling schedule is thin in the file; the only direct figure is SimGen's +4.4 mIoU, on a different task. So this alternative is a real threat. The comparison against the rerun FREDOM loss and DAFormer with Rare Class Sampling settles it. If filtered generation adds nothing when stacked on either, the idea fails.
Why not just filter with the model's own confidence, as Curated LLM does? That is exactly the region-level class-blind ablation. The argument against it is structural: rare classes sit at low confidence throughout training, so a global threshold removes them first. The per-class retention analysis of hand-verified correct regions tests this directly. If class-blind filtering retains correct rare regions as well as ours does, the class-relative reference is unnecessary and we will report that.
Why not just check generated images with a pretrained open-vocabulary segmenter and drop regions where it disagrees with the label? This is a strong, cheap alternative, and it may work comparably. It also brings in outside knowledge that the learning-dynamics filter does not use, and it has its own rare-class blind spots. We will run it as an additional baseline with the same ignore-masking and threshold tuning. If it matches our filter, the contribution reduces to the finding that region-level, class-aware curation is what matters, whatever the scorer. We would then frame the result that way and not claim the dynamics signal is special.
Why not just fix the generator, as Driving with DINO does? Better conditioning reduces mismatches but, as that paper itself frames the dilemma, it does not eliminate them for high-level layouts. Our filter is independent of the generator and can sit on top of any of them. We test this by also running the filter on a pool produced with a stronger, edge-conditioned setting. Our method should help less there, but still help on the omitted small objects.
Proposed Experiments and Decisive Checks
The main experiments are GTA5→Cityscapes and SYNTHIA→Cityscapes. We report mean IoU and per-class IoU on Cityscapes validation, averaged over three seeds, with separate means for rare and common classes.
The first decisive check is the key ablation. The class-relative filter must beat region-level class-blind filtering on rare-class IoU while matching it on common classes. The retention analysis should show the mechanism: similar rejection of corrupted regions, and a higher keep-rate for correct rare regions.
The second check measures the filter against ground truth. On injected corruptions and the hand-audited sample, we report the precision and recall of rejection per class. A filter that improves IoU without detecting real mismatches would suggest the gain comes from something else, such as ignoring hard pixels, and would undercut the claim.
The third check is a dose-response test. We vary the number of pasted rare instances per class. If curation is doing its job, filtered augmentation should improve monotonically with more generated instances, whereas unfiltered augmentation should saturate or decline. This echoes the oversampling paradox of Bansal et al..
The fourth check tests whether generated rare-class data hurts common classes, the side effect reported by Das et al. We track common-class IoU and confusion between each rare class and its visually nearest common class, such as bus versus truck and rider versus person.
As an out-of-domain sanity check, we apply the same image-level variant of the class-relative filter to the Caltech Camera Traps deer setting of Domain Adaptation for Rare Classes Augmented with Synthetic Samples. We ask whether filtering CAD-rendered deer against real deer confidence helps beyond the reported DeerCORAL gain of 22.4% in deer accuracy. That tells us whether the idea carries beyond driving segmentation.
References
- Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
- FREDOM: Fairness Domain Adaptation Approach to Semantic Scene Understanding
- SimGen: Simulator-conditioned Driving Scene Generation
- Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Driving
- ADAPTATION OF DOMAIN-SPECIFIC TRANSFORMER MODELS WITH TEXT OVERSAMPLING FOR SENTIMENT ANALYSIS OF SOCIAL MEDIA POSTS ON COVID-19 VACCINE
- Emancipating Event Extraction from the Constraints of Long-Tailed Distribution Data Utilizing Large Language Models
- Domain Adaptation for Rare Classes Augmented with Synthetic Samples