Context-Conditioned Inpainting for Pedestrian Hazard Detection

A person walking with a camera-equipped cane or smart glasses needs to be warned about the uplifted paving slab, the missing drain grate, or the scooter lying across the path. They also need not to be warned about the ordinary change from brick to asphalt, or about the tactile paving that marks a curb ramp. A standard way to find unexpected objects is reconstruction-based anomaly detection. A generative model is trained only on "normal" images. It tries to redraw each new image, and wherever it fails to redraw what is there, something unusual is present. The trouble is that these detectors assume one fixed idea of "normal," usually a single industrial texture or highway asphalt. On a sidewalk, normality depends on the situation. The same studded tile is expected at a platform edge and odd in an office corridor. A change of pavement material is not a hazard, but a gap in it is.

We propose to make the reconstruction depend on context. A lightweight segmentation network produces a coarse map of surface material and walking zone, meaning the part of the ground the path runs over. A small inpainting network then hides each ground patch and regenerates it. It sees only the patch's surroundings and that context map, never the hidden pixels themselves. The anomaly score is the difference between the regenerated patch and the real one, measured in feature space. The claim is that conditioning on situational context removes false alarms at normal environmental transitions. It also lets identical texture be scored differently in different functional zones. Hiding the patch removes the "identical shortcut," where a decoder with enough capacity simply copies the anomaly through. If both hold, one model can work across mixed pedestrian terrain at edge-device speed.

Core Claim and Where It Sits

The closest existing work is JSR-Net. It attaches a narrow-bottleneck reconstruction module to a frozen segmentation network and trains it to reconstruct only drivable road. It combines the structural-similarity residual with segmentation logits and reports 94.4% average precision on Road Anomaly and 0.4% FPR95 on Road Obstacles. JSR-Net supplies the method we reuse: a module that sits on a frozen backbone and is trained with synthetic pasted anomalies. It differs from our proposal in three ways. It reconstructs a region from its own compressed pixels, which is exactly the channel through which small anomalies leak back into the output. Its idea of normal is a single road class. Segmentation enters only after reconstruction, when the two scores are merged, rather than shaping what the reconstructor expects to see. Our proposal moves context upstream, into the conditioning of the generator.

A second nearby line handles context shift discriminatively. Residual Pattern Learning with Context-robust Contrastive Learning trains a residual head on a frozen segmenter and aligns anomaly embeddings across backgrounds so that moving from city to countryside does not raise false alarms. It reports 71.60% area under the precision-recall curve on Road Anomaly. This paper motivates our work, because it shows context shift is a real source of false positives. It also competes with it, because it tries to make the detector ignore context. We instead make normality a function of context. That is the only route to the tactile-paving case, where the right answer changes with the zone.

Safe Robot Navigation Via Multi-Modal Anomaly Detection models safe terrain with a normalizing flow over RGB, depth and surface-normal features, reaching above 95% AUROC. It gives a likelihood score but no redrawn surface, and it has no notion of zone. VisionGPT does the opposite. It uses a large language model to switch an open-vocabulary detector's class list between contexts such as street and park, and splits the frame into spatial zones with a forward ground zone for trip hazards. It reports 88.01 zero-shot AP and 60 ms latency on its own first-person clips. VisionGPT supplies our zoning idea and our strongest "just prompt it" competitor. It cannot see hazards that have no nameable object boundary, such as a drop-off or a missing slab, and our proposal targets exactly those.

The hiding design responds to a constraint. ReDi shows that high-capacity decoders learn to copy defects through. It avoids this by reducing inputs to gradient histograms and retrieved normal exemplars, reaching 98.8% detection AUROC on MVTec AD. Hiding the patch is a different fix for the same problem. It keeps full texture resolution in the surroundings, which the sidewalk materials catalogued in CitySurfaces require, while guaranteeing that the target pixels never reach the generator.

Research Program

The context network is an HRNet-style material segmenter trained on CitySurfaces, which reports 90.5% test mean intersection-over-union. Its output is extended with a small set of functional-zone labels: open sidewalk, curb ramp or platform edge, crossing, and indoor floor. The zone labels are derived from OpenStreetMap-style annotations where available, following the location tagging in FootNotes.

The context map is deliberately blurred and majority-pooled over large regions before it reaches the inpainter. This matters because a small anomaly should not be able to relabel its own context. If the segmenter calls a pothole "asphalt," that is the desired behaviour, since asphalt is what should be there. If the segmenter carves the pothole out as its own region, pooling restores the surrounding label.

The inpainter is a single-pass convolutional network, not an iterative diffusion model, so that it meets wearable latency. It works on a grid of ground patches. For each patch it receives the surrounding ring of real pixels plus the pooled context code, and it predicts the hidden centre. Anomaly localisation needs only the ground region, so hidden patches can be batched in one forward pass.

Let $x_p$ be the true patch, $\hat{x}_p$ its inpainted estimate, and $\phi$ a frozen pretrained feature extractor. The per-patch score is

$$s_p = \lVert \phi(x_p) - \phi(\hat{x}_p) \rVert_2 \, / \, \sigma_{c(p)},$$

where $\sigma_{c(p)}$ is the typical inpainting error measured on held-out normal data for the context class $c(p)$ at that patch. This normalisation stops coarse cobblestone from being penalised relative to smooth concrete. Comparing in feature space rather than pixel space addresses the sensitivity of pixel residuals to shadows and wetness.

Following DRÆM and JSR-Net, a small scoring head is then trained on synthetic anomalies pasted onto the ground. Its inputs are $s_p$, the zone code and image position. Two kinds of synthetic anomaly are used. The first is Perlin-noise blobs and polygons. The second, more importantly, is context-swap anomalies: real normal texture from one zone pasted into another, such as tactile paving placed on corridor floor. The second kind is what teaches the head that an anomaly can be a real surface in the wrong place.

Position enters the head because egocentric frames have systematically higher error near the horizon and edges, and VisionGPT's zoning shows that forward ground hazards matter most.

Baselines

The naive reference point is MaxLogit on the same frozen segmenter, with no extra training. Hendrycks et al. report 26.5% FPR95 for it on StreetHazards, against 33.7% for maximum softmax probability. It will be computed on every test set from the exact backbone we use. Our method must beat it by a clear margin, especially at material boundaries, where logit methods are known to fire.

The strongest existing reconstructive method is JSR-Net. We will retrain it with the authors' recipe on the same normal pedestrian training images and the same synthetic polygon anomalies. It gets the same backbone and the same tuning budget, measured in GPU-hours of hyperparameter search. We will also evaluate it on its native road benchmarks, where its published figures are 94.4% AP on Road Anomaly and 0.4% FPR95 on Road Obstacles. On road data, beating it means at least matching those figures. That is not our main target; not losing more than a few points is acceptable. On pedestrian data, beating it means higher AP and lower FPR95, with the gap concentrated at material transitions and on low-profile hazards.

The strongest discriminative competitor is RPL with Context-robust Contrastive Learning, retrained identically. Its reported 71.60% AuPRC on Road Anomaly is the reference.

The open-vocabulary competitor is a VisionGPT-style pipeline: YOLO-World with context-switched hazard vocabularies. The LLM reasoning stage is removed so that we score only detection. Its published 88.01 AP is on its own dataset, and we will report it only as context.

Two ablations isolate the key ingredient. The first is the identical inpainter trained with the context code set to a constant. It tests whether conditioning matters at all. The second is a bank of separate inpainters, one per context class, with hard switching on the pooled label. It tests whether a per-pixel shared model is better than simply partitioning the data.

A third ablation replaces hiding with a narrow bottleneck on the patch itself. It separates the shortcut fix from the context effect.

Why not just…?

Why not just train one reconstructor per surface type and switch between them? This is the most serious threat. If context labels are reliable and scenes contain one material at a time, hard switching could match our model. We expect it to fail in two places. Transition regions straddle two models, and neither has seen the boundary. Rare contexts, such as a single class of curb ramp, lack the data to train a separate model. The hard-switch ablation settles this directly. If it matches the shared model on transition-heavy test frames, we will report that the gain comes from context, not from sharing, and adopt the simpler design.

Why not just make the detector context-invariant, as RPL does? Invariance is the right goal when context is a nuisance, and RPL's reported 71.60% AuPRC shows it works for that purpose on roads. It cannot represent a surface that is anomalous only in a particular place. The context-swap test below measures this. If RPL scores near chance on it while our model does not, the distinction is empirical and not rhetorical.

Why not just prompt an open-vocabulary detector or a vision-language model for "hole" or "broken pavement"? Detectors trained under a closed-world assumption assign unfamiliar debris to inlier classes with confidence, as the Fishyscapes study found (Blum et al.). Full vision-language descriptions took about 4.4 seconds in the wearable system of Chen et al., which is too slow for a trip warning. The YOLO-World baseline tests the fast version of this idea. We expect it to do well on nameable objects such as scooters and cones and poorly on boundary-less defects, and we will report results split by hazard type so this is visible.

Why not just fit a ground plane from depth? Plane fitting handles tall obstacles well. Pinggera et al. show global planar assumptions break on irregular profiles and curb transitions. Wellhausen et al. show geometry alone cannot distinguish a leaf pile from a rock. Where depth is available we will run a RANSAC plane baseline. If it wins on protruding obstacles, the honest outcome is a hybrid, with our model covering flat-texture defects that geometry misses.

Proposed Experiments

There is no public pedestrian-viewpoint reconstruction-anomaly benchmark in our sources, which is itself a gap. We will therefore build a test set. Training data is normal sidewalk imagery from CitySurfaces plus our own chest- and cane-height egocentric video. Testing uses held-out egocentric clips with pixel-annotated hazards, split into three strata: protruding objects, flat surface defects and drop-offs, and context-swap cases. For comparability we also report on Road Anomaly, Road Obstacles, Lost and Found and Fishyscapes. For robustness under exposure and weather we report on WildDash, where a Cityscapes model fell from about 64 to 16.5 mIoU. The metrics are pixel AP, FPR95 and AUROC, with false positives per minute reported separately inside a band two segmentation cells wide around material transitions. Latency is measured on a phone-class GPU.

Three results would decide the idea.

The first is transition false positives. On frames containing material changes, the context-conditioned model must show substantially fewer false positives than both the constant-context ablation and JSR-Net at equal recall. If it does not, the central claim fails.

The second is context-swap discrimination. We take real tactile-paving and similar textures and score them in their normal zone and in a foreign zone. The model must separate the two cases, and the invariant baselines should not. Without this, conditioning buys only smoother false-positive rates, not new capability.

The third is shortcut suppression. On small low-profile hazards, the hidden-patch inpainter must beat the self-bottleneck ablation in recall without raising false positives on coarse textures. If every variant misses thin or reflective hazards, as JSR-Net and Wellhausen et al. both report for their settings, we will state that limit plainly rather than tune around it.