Simulators such as CARLA produce unlimited driving scenes with perfect labels, including the question-answer and description text needed to train vision-language models. Models trained only on this data still fail on real camera footage. The usual fix is to make the model's internal image features look the same for simulated and real inputs. We propose a different place to intervene. We learn a scorer, a small model that rates how good a generated description or answer is for an image. The scorer is trained on simulated data, where the true quality of any output can be computed from the simulator's labels. It is forced to be unable to tell simulated from real images. Then the generator is trained directly on unlabeled real images, using that scorer as its reward.
The training method for the generator is self-critical sequence training (SCST). This is a reinforcement-learning recipe for text generators. The model samples an output, compares its score against the score of its own greedy output (the most likely word at every step), and is pushed toward samples that beat that baseline (Self-Critical Sequence Training for Image Captioning). SCST normally needs human reference text to compute its reward, and unlabeled real images have none. Our proposal supplies that missing reward through the simulator, while keeping the policy's own trials on real pixels. The bet is that this pays off exactly where sim-to-real gaps bite: the generator never has to generalize across the gap, only the scorer does.
Core Claim
A reward model can be made to transfer from simulation to reality by adversarial domain adaptation. Adversarial domain adaptation means training a feature extractor to fool a classifier that tries to guess which domain an input came from. Once the reward model transfers, SCST on unlabeled real images with that reward will beat two alternatives on real-domain factual accuracy: generators adapted by feature alignment, and generators trained with frozen, off-the-shelf reference-free rewards.
The claim can fail in a specific way. SCST actively searches for outputs that score well. If the scorer is miscalibrated on real images, the generator will find and exploit that miscalibration. So the method stands or falls on whether domain-invariant training keeps the scorer's rankings honest on real data under optimization pressure, not merely on a fixed test set. We treat measuring that "reward-hacking gap" as a first-class result.
Why This Is Not Already Done
Three bodies of prior work come close, and each takes a different route.
Show, Adapt and Tell adapts a captioner to an unlabeled domain using two discriminators as reward: one judges whether text looks like target-domain text, and one judges whether an image and a caption match. It trains with Monte Carlo policy gradients. Its critics are trained only on target data and never inherit source-side ground truth. Its shifts are real-to-real, such as MSCOCO to CUB-200. DiscriTune gets reference-free rewards from a frozen CLIP retriever, rewarding captions that let CLIP pick the right image out of 100. That reward measures discriminativeness, not correctness, and nothing about it uses simulator labels. CoC-VLA does bridge CARLA and nuScenes. It adversarially aligns a simulation-trained teacher's internal features with a real-data student, trained with supervision and no sequence-level reward.
None of these adapts the reward model across the sim-to-real boundary and then runs self-critical training on real images. Swapping Monte Carlo rollouts for a greedy baseline inside Show, Adapt and Tell would be a small engineering change. We regard it as a baseline, not the contribution.
Research Program
The scorer. The scorer $R_\phi(x, w)$ rates an output text $w$ for an image $x$. It is a frozen pretrained image-text encoder topped with a trainable fusion head. Its targets come from the simulator. For a simulated image with reference text $y^*$ and scene metadata, the target $r^*(x,w)$ combines two things. The first is a metric score, such as CIDEr (a word-overlap score against reference text) of $w$ against $y^*$. The second is a factual-consistency score, computed by checking object presence, counts, and left/right and ahead/behind relations stated in $w$ against the simulator's object list.
To teach the scorer what wrong looks like, we build negatives by editing correct outputs. We swap spatial relations, change counts, and substitute objects. This targets the spatial weakness that RLS3 documents in CLIP (50% baseline spatial accuracy).
The scorer's image-side features $h_\phi(x)$ also feed a domain classifier $D$ through a gradient reversal layer. This layer passes features forward unchanged but flips the sign of the gradient on the way back, which pushes the features toward domain invariance. The training objective is
$$\min_\phi \max_D \; \mathbb{E}_{x \sim \text{sim}}\big[(R_\phi(x,w) - r^*(x,w))^2\big] - \lambda\, \mathcal{L}_{\text{dom}}\big(D(h_\phi(x_{\text{sim}})), D(h_\phi(x_{\text{real}}))\big).$$
Here $\mathcal{L}_{\text{dom}}$ is the domain classifier's cross-entropy on simulated versus unlabeled real images, and $\lambda$ trades off regression fit against invariance.
Shortcut risk. ASM showed that domain classifiers in sim-to-real LiDAR latch onto low-level sensor artifacts. That paper constrains our design. We therefore attach the classifier only after the frozen encoder, which is less sensitive to rendering noise. We also report a probe that measures how much domain information remains in $h_\phi$.
The generator. A pretrained vision-language generator is first fine-tuned with cross-entropy on simulated data. It is then trained with SCST on unlabeled real images:
$$\nabla_\theta L \approx -\big(R_\phi(x, w^s) - R_\phi(x, \hat w)\big)\nabla_\theta \log p_\theta(w^s \mid x).$$
Here $w^s$ is a sampled output and $\hat w$ is the greedy output. A cross-entropy term on simulated data and a sampling-diversity check guard against the policy collapse and "bag-of-words" degeneration reported for discriminative rewards (Discriminability Objective for Training Descriptive Captions).
The scorer is periodically refreshed with fresh generator samples on simulated images. This lets it see the kinds of outputs the policy now produces.
Regimes. The main setting is driving question answering and scene description, from DriveLM-CARLA (1.6M QA pairs, per DriveLM) to real nuScenes-VLM / DriveLM-nuScenes. Real text references are used only for evaluation and for a small held-out calibration set. A second setting tests generality in the classic captioning regime. There, the synthetic multi-context training images of ICSD serve as the labeled source, and unlabeled MSCOCO and Flickr30k images serve as the target. This setting lacks structured scene metadata, so the scorer target reduces to the metric score plus caption-swap negatives. That is a useful test of how much the factual term matters.
Baselines
Do nothing. The simplest reference point is the generator fine-tuned with cross-entropy on simulated data and applied to real images unchanged. It shows how large the gap is. A second, stronger naive reference adds standard SCST on simulated data with CIDEr against simulator references. This separates what SCST buys in-domain from what transfer buys.
CoC-VLA. The strongest existing method on the driving question is CoC-VLA. On nuScenes-VLM after CARLA transfer it reports CIDEr 23.21 (versus 3.62 for DriveLM), ADE 1.44, and a 1.58% collision rate. It is the strongest competitor because it uses the same simulated and real sources and also relies on adversarial alignment, only on the generator's features rather than on the reward. We will rerun it on our backbone with the same simulated data, the same unlabeled real images, and the same GPU-hour and hyperparameter-search budget, and also report its published numbers.
Beating CoC-VLA means higher CIDEr and higher factual-consistency accuracy on nuScenes-VLM on matched backbones. We also require that our method does not worsen ADE or collision rate if trajectory tokens are generated. If our rerun of CoC-VLA falls far below 23.21, we will report both numbers and not claim a win against the published figure.
Captioning setting. The reference is ICSD's reported CIDEr of 96.6 on MSCOCO and 54.3 on Flickr30k when trained only on synthetic data. Our method must exceed ICSD started from the same model and trained with the same compute.
DiscriTune-style reward. The strongest reference-free reward competitor is SCST on the same unlabeled real images with a frozen CLIP retrieval reward, in the style of DiscriTune. That paper reports raising ClipCap P@1 on MS COCO from 74.2% to 84.8%. We run it with an identical greedy baseline, identical sample counts, and identical compute.
Ablations. The key ablation removes gradient reversal: the same scorer trained on simulated data without domain adaptation, then used for real-image SCST. If this matches the full method, the adversarial ingredient is unnecessary and the claim fails. A second ablation drops the factual term from the scorer's target. A third reproduces Show, Adapt and Tell's target-only critics but trains them with a greedy self-critical baseline instead of Monte Carlo rollouts.
Why not just…?
Why not just use a frozen CLIP retrieval reward on real images, as DiscriTune does? This is the most serious threat, because it needs no simulator and already works cross-domain. Retrieval rewards any detail that separates an image from distractors, not details that are true. Its ranking of spatial relations is only as good as CLIP's, and RLS3 reports that is weak. We expect our method to win on factual-consistency and spatial-relation accuracy, and possibly to tie on generic CIDEr. If DiscriTune matches us on factual accuracy in the driving setting, the simulator-derived reward adds nothing and we will say so.
Why not just align the generator's features, as CoC-VLA does? Feature alignment asks the whole generator to become domain-invariant, and ASM shows how such classifiers fall back on sensor artifacts. Our approach only asks a small scorer to be invariant, while the generator trains on real pixels. The head-to-head on nuScenes-VLM, plus the domain probe on both models' features, settles which is easier to make invariant.
Why not just translate simulated images to look real, as PixelDA does, and then train as usual? PixelDA handles style and lighting shifts well. It cut the LineMod angle error to 23.5°. But it is not designed for layout or geometry, and multi-object driving scenes differ from simulation mostly in those. We will include a translated-image SCST baseline in the captioning setting, where style dominates. There translation is a genuine competitor, and a tie there would narrow our claim to structurally complex scenes.
Why not just skip the adversarial term, since the frozen encoder may already be invariant? This is exactly the no-reversal ablation. We report it on both the scorer's calibration and the downstream real-domain metrics.
Proposed Experiments and Decisive Checks
The primary metrics on nuScenes-VLM and DriveLM-nuScenes are CIDEr, factual-consistency accuracy, and ADE and collision rate where trajectories are produced. For factual consistency, a held-out set of about 1,000 real frames will be manually annotated with object and relation facts; no labeled real frames are reported for this in the prior work. The captioning setting reports CIDEr and SPICE on MSCOCO and Flickr30k, plus human fluency ratings. These ratings test the fluency tension raised by the Discriminability and DiscriTune results.
The decisive check is the reward-hacking gap. During real-image SCST, we track the scorer's predicted improvement alongside the true improvement measured on the labeled real calibration set. The method succeeds only if, relative to the no-reversal scorer, the adversarially adapted scorer shows three things:
- Calibration before training. A higher rank correlation with the true reward on real images before SCST begins.
- Honesty under optimization. A smaller gap between its predicted and true improvement after SCST.
- Downstream gains. Higher final factual accuracy.
If the gap is equally small for both scorers, the adversarial ingredient is not needed. If both gaps are large, reward transfer is not viable at this scale.
A secondary check varies $\lambda$ and plots scorer calibration against the domain probe's accuracy. This tests whether invariance, rather than extra regularization, is what drives the effect.
References
- Self-Critical Sequence Training for Image Captioning
- Show, Adapt and Tell: Adversarial Training of Cross-Domain Image Captioner
- Cross-Domain Image Captioning with Discriminative Finetuning
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Discriminability Objective for Training Descriptive Captions
- DriveLM: Driving with Graph Visual Question Answering
- Image Captioning with Multi-Context Synthetic Data
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks