A household robot or street-level assistant is told to "plug in the kettle" or "cross here." The instruction is harmless, but the scene may not be: a frayed cable lies near standing water, or a manhole cover is missing at the curb. Systems that check proposed actions against what the camera sees, called safety guardrails, are now trained largely on synthetic images, because real hazardous scenes are rare and dangerous to stage. This proposal argues that these guardrails have a specific weakness. Their training images come from a narrow slice of the world, and the detail that makes a scene dangerous is often thin and small: a wire, an edge, a gap. Image generators are poor at drawing such details faithfully. A guardrail trained this way may learn the generator's style or the look of one region rather than the hazard, and it fails when deployed somewhere else.
We propose hazard twins. Each twin is a pair of images that differ only in a small, hazard-defining region and are placed against real backgrounds drawn from many geographic regions. A dense segmentation model then checks each pair at the pixel level: the hazard detail must actually be present in the risky image, absent in the benign one, and the rest of the image must be unchanged. Pairs that fail the check are discarded. The claim is that pixel-verified pairs with diverse regional backgrounds produce guardrails that hold up in geographic regions never seen in training. They should do better than simply generating more synthetic data, adding region names to the generation prompts, or fine-tuning on a small set of real images from many regions.
Core Claim
The most closely related existing system is EMBGuard. It trains a 2B–4B-parameter guardrail on 15.1K synthetic (observation, proposed action) pairs, built by editing scene graphs with a large language model and a text-to-image diffusion model. A diffusion model is a generator that learns to turn random noise into images step by step. EMBGuard reports a Potential Risk score of 71.2% on its held-out set. Its data, however, is entirely indoor and domestic. It also never checks whether the generated hazard is visually present at the resolution where it matters. We make two claims that EMBGuard does not test.
The first is about failure under regional shift. When a guardrail moves to a new geographic region, its errors should concentrate on hazards defined by small, thin structures. Coarse hazards, such as a fire or a large blade, should transfer comparatively well. Two findings make this plausible. Vision-language models fall to near chance when a judgment depends on fine geometry, as on the Unreal Engine minimal-pair benchmark in Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video. Separately, a regional shift changes exactly the textures and layouts (wiring practice, curb design, building materials) against which thin hazards must be picked out, as Cheng et al. and Astruc et al. document for remote sensing.
The second is about the training data. Pixel-verified twins with diverse regional backgrounds should close most of that gap, and verification, not volume, should be what does the work. 3D MCGAN found that doubling a detection set with synthetic lesions raised the competition performance metric (CPM) from 0.518 to 0.550, but tripling it hurt, because the detector began fitting generator artifacts. We read that result as a warning that unfiltered synthetic volume teaches shortcuts. The twin construction is designed to remove those shortcuts. Because the two images in a pair differ only inside the hazard mask, any cue the guardrail can exploit has to come from the hazard itself.
Research Program
The pipeline starts from real photographs, indoor and street-level, collected under permissive licenses from a set of source regions. Several further regions are held out entirely. The 360CityArena footage of Akihabara (360CityArena) is one natural held-out region, since its frames are real urban imagery from a single district.
For each background, a language model proposes a (hazard, action) scenario adapted to that scene. Following EMBGuard's taxonomy, each scenario is tagged as causal-risky (the action triggers the hazard), selective-risky, decoupled-benign (a hazard is present but the action does not engage it), or absent-benign. A diffusion inpainting model, which redraws only a masked region of an existing image, then paints the hazard into a specified mask. A second inpainting pass inside the same mask produces the benign twin, either repairing the hazard or leaving it in a place the action does not touch. This pairing addresses the over-conservatism EMBGuard reports, where guardrails refuse safe actions just because a hazard is visible somewhere in the scene. The benign twin shows the same hazard in a non-engaging position, so the guardrail cannot learn "hazard visible means refuse."
Verification is the step that separates this pipeline from earlier ones. We run gen2seg, which repurposes a generative model's full encoder-decoder for instance segmentation and is unusually good at thin structures (51.4 mIoU on iShape against SAM's 16.8). A pair is accepted only if three conditions hold. First, the segmented hazard instance in the risky image overlaps the intended mask above a threshold. Second, no comparable instance appears there in the benign image. Third, the two images agree outside the mask up to inpainting noise. This turns spatial detail preservation from a model-architecture concern into a filter on what data is allowed into training.
The guardrail itself is a small vision-language model. It is trained on accepted pairs with the usual safety-label and explanation targets, plus two additional signals. One is an auxiliary dense head that predicts the hazard mask; the edit mask comes free from generation. The other is a paired ranking loss that requires the risky twin to score as more dangerous than its benign partner. The mask head sits on skip-connected high-resolution features. We use that design because Fine-tune Smarter, Not Harder found that keeping a U-Net-style skip decoder preserved dense accuracy on geographic hold-out sets.
Baselines
The naive reference point is a frontier vision-language model asked to check for hazards before approving an action, with no further training. The best number in the file for this setting is Claude 3.5 Sonnet at 64.0% average accuracy on MSSBench (Multimodal Situational Safety). We run two open models of comparable size to our guardrail under an identical prompt, and report Claude or GPT-class results where API access allows.
The strongest prompting baseline is the multi-agent decomposition from the same paper, which splits the job into a visual describer, an intent reasoner, and a safety judge. With ground-truth environment states it raised Claude's embodied-task accuracy from 62.1% to 83.6%. We run it on raw images only, without ground-truth states, because that is the deployment setting.
The strongest trained competitor is EMBGuard-4B, which reports a Potential Risk score of 71.2% on its held-out set. For GPT-5.1 and Gemini-2.5-Pro, the file gives no reported numbers on EMBGUARDTEST. We retrain the EMBGuard recipe on the same backbone, with the same number of training pairs and the same tuning budget as our model, so that only the data construction differs.
The ablations isolate the key ingredients. The first gives EMBGuard-style generation region-diverse backgrounds but no verification. The second adds verification but uses backgrounds from source regions only. The third removes the mask head and the ranking loss. Beating the field means two things. On held-out regions, our model must improve risky-action recall while keeping the benign false-refusal rate no higher than the strongest baseline's. And on the small-detail subset, it must cut the source-to-held-out gap by at least half relative to the retrained EMBGuard.
Why not just…?
Why not just add region names to the diffusion prompts and generate more EMBGuard-style data? This is the most direct threat, and it is our first ablation. The MCGAN saturation result (3D MCGAN) and gen2seg's evidence that standard pipelines lose thin structures both suggest that unverified generation will often draw a regional style but miss the hazard detail, producing pairs whose labels are wrong. If prompt diversity alone matches verified twins on the held-out small-detail subset, verification is unnecessary and we will report that.
Why not just fine-tune on a small set of real images from many regions? Banerjee et al. show that a small, ecologically diverse training set can beat a larger one drawn from regions near the target. Low-rank adaptation (LoRA), which updates only small added weight matrices, outperformed full fine-tuning by more than 5 points on the Sen1Floods11 hold-out (87.57% vs. 82.07% mIoU) (Fine-tune Smarter, Not Harder). That makes this a strong competitor. The obstacle is that real hazardous scenes, and especially matched benign twins, are rare and costly to label. We therefore compare at equal annotation cost: N real labeled images from diverse regions with LoRA, against verified twins from the same budget spent on unlabeled backgrounds. We also test the combination. If real-diverse LoRA wins outright, the contribution shrinks to a cheaper approximation of real data, and we will say so.
Why not just feed the guardrail higher-resolution images or tiled crops? Higher resolution addresses perception, but it cannot fix training data in which the hazard was never rendered correctly. MSSBench also shows that models often describe the hazard in their own captions and still approve the action, so seeing the detail is not the same as acting on it. We include a high-resolution tiling baseline trained on unverified data. If it closes the small-detail gap on its own, the bottleneck is input resolution, not data, and the verification step should be dropped.
Proposed Experiments
The first experiment is diagnostic and gates everything else. We build a real held-out test set of roughly 300 curated scenarios per region, matching the scale of EMBGUARDTEST's 329, across at least four held-out regions including Akihabara. Each scenario is labeled by hazard area relative to image size. We then test whether the retrained EMBGuard's errors under regional shift concentrate in the small-detail stratum. If they do not, the spatial-detail part of the proposal is wrong, and the project reduces to region diversification.
The main experiment trains every baseline and ablation at matched pair counts and evaluates them in three settings. The held-out-region set is the primary target. EMBGUARDTEST and the MSSBench embodied split check that in-domain performance does not degrade. The Unreal Engine minimal-pair videos (Assessing Situational and Spatial Awareness of VLMs), using single frames or short clips, probe fine spatial judgment.
We report four quantities: risky recall, benign false-refusal rate, the per-region gap, and the small-detail gap. A synthetic-to-real ratio sweep at 1×, 2×, and 3× tests whether verified twins avoid the decline MCGAN observed. A shortcut probe trains a classifier to distinguish generated images from real ones using the guardrail's features. If verification succeeds in removing generator cues, that classifier's accuracy should be lower for our model than for the unverified ablation.
The decisive result is three-part. Verified twins must beat unverified region-diverse twins at equal count on the held-out small-detail stratum, without raising false refusals. They must match or beat real-diverse LoRA at equal annotation cost. And their advantage should grow, not shrink, as the synthetic ratio increases.
References
- EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agents
- Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video
- Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Review
- OmniSat: Self-supervised Modality Fusion for Earth Observation
- Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-Based CT Image Augmentation for Object Detection
- 360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents
- gen2seg: Generative Models Enable Generalizable Instance Segmentation
- Fine-Tune Smarter, Not Harder: Parameter-Efficient Fine-Tuning for Geospatial Foundation Models
- Multimodal Situational Safety
- Trusting small training dataset for supervised change detection