Context-Conditioned Denoising Scores for Open-Vocabulary Road Anomaly Segmentation

A self-driving car's segmentation network sorts every pixel into about twenty known classes. When it sees something outside that list, such as a fallen crate or an animal, it confidently calls it "road" or "car." The standard fix is to add an anomaly score. The better scores combine two signals: how unlikely a pixel's appearance is under a model of normal data, and how poorly it fits any known class. Two things are missing. The first is a likelihood signal that asks "is this plausible here, given the rest of the scene?" rather than "is this local texture unusual?" The second is a class-fit signal that is not tied to a fixed list of categories. Diffusion models are the obvious tool for likelihood: they generate data by learning to undo added noise, and how well they denoise tells you how typical an input is. Existing diffusion anomaly detectors, however, run long iterative inpainting loops that are far too slow for a moving vehicle.

We propose to train a small diffusion model on the frozen feature maps of a vision-language backbone, meaning a network whose per-pixel features live in the same space as text embeddings. The model is trained only on normal driving scenes. Its denoising error is read at two noise levels in a single forward pass, with no sampling. At low noise, the error measures whether local features look normal. At high noise, the denoiser can only succeed by predicting each location from the rest of the image, so its error measures whether the content belongs in this scene. That second signal only exists if the denoiser can see the whole scene. We therefore give it criss-cross attention, a cheap way to connect every location to every other. We fuse the two denoising errors with a text-based "fits a known class" score, and calibrate the combination statistically rather than with hand-tuned weights. Because the features share a space with text, a flagged region can also be named afterward.

Core Claim

The blind spot of generative anomaly scores is well known: they reward inputs that share low-level statistics with normal data. Our claim is that in dense prediction this blind spot is partly a receptive-field problem. A denoiser that sees only a small neighborhood can only model local texture density, so it inherits exactly that blind spot. A denoiser with scene-wide context, evaluated at high noise, turns into a contextual plausibility test.

We make three testable predictions:

  1. Road anomaly average precision should rise as the denoiser's receptive field grows, from stacked 3×3 convolutions, to local windows, to criss-cross attention, to full attention.
  2. The gain should come from the high-noise term, not the low-noise term.
  3. The whole system should run in one denoiser pass per noise draw on a low-resolution feature grid, orders of magnitude cheaper than iterative inpainting.

Relation to Prior Work

Three papers motivate the design and supply its parts.

DenseHybrid shows that fusing an implicit likelihood with a discriminative in-distribution probability works for dense road anomalies. It computes the likelihood as a log-sum-exp over classifier logits and forms a log-ratio with the discriminative probability. That likelihood is defined only over a fixed class set and is computed from features whose context is whatever the segmentation backbone happens to provide. We replace it with an explicit denoising density whose context we control.

MadSGM supplies the scoring idea. It fuses reconstruction error, an exact probability-flow likelihood, and the norm of the score function (the gradient of log-density with respect to the input) for multivariate time series. It uses 1D convolutions and has no spatial context question. We borrow its score-norm term and move it to 2D scenes with a variable receptive field. We deliberately drop its exact likelihood integration, which requires many solver steps.

GROOD supplies the calibration method. It fuses a semantic-shift score and a distance-based score under a 2D Gaussian Neyman-Pearson test, which yields thresholds at a chosen false-negative rate. It operates only on whole-image embeddings. We apply its calibration per pixel.

The closest existing system is DiffuGTS. For medical tumors it combines text-prompted attention maps from a frozen diffusion model, pseudo-healthy latent inpainting, and a pixel-plus-feature residual. The combination of diffusion, open vocabulary and hybrid residual is therefore already done. DiffuGTS reports roughly 60% to over 63% Dice from inpainting on unseen liver tumors, and its authors state that inference is slower than U-Nets. Our proposal differs in three ways. We use no inpainting or sampling at all. We treat the denoiser's receptive field as the experimental variable. And we need no anomaly prompt: the text term encodes only the known classes.

OVDiff also uses diffusion for open-vocabulary segmentation, by synthesizing 32 support images per query. It is a segmenter, not an anomaly detector, and its cost is exactly what we avoid.

On architecture, CCNet provides the context operator. Two recurrent passes of row-and-column attention connect every location to every other, with 11× less GPU memory and 85% fewer FLOPs than non-local blocks. SAN constrains the design in a useful way. It shows that a heavy frozen CLIP model can run at low resolution while a light high-resolution side network restores boundaries. We use the same split to recover sharp anomaly masks from a coarse denoising map.

Research Program

The backbone is a frozen dense vision-language encoder. Our default is the frozen CLIP branch of SAN, and we will also test a dense CLIP model that has been self-distilled for better per-pixel features. It produces a feature map $f$ at 1/16 resolution; for a 1024×2048 frame this is a 64×128 grid, about 8,000 positions.

On Cityscapes training frames only, we train a denoiser $\epsilon_\theta$ with the standard objective. We add Gaussian noise $\epsilon$ at level $\sigma$ to the features and train the denoiser to predict that noise from the noisy features $z_\sigma = f + \sigma\epsilon$. The denoiser is a small network of a few million parameters. Its context operator is swappable, which is the core experimental knob.

At test time we draw $K$ noise samples, with $K$ between 1 and 4, at a low level $\sigma_\ell$ and a high level $\sigma_h$. The per-location denoising error at level $\sigma$ is

$$ e_i(\sigma) = \frac{1}{K}\sum_{k=1}^{K} \big\| \epsilon_\theta(z^{(k)}_\sigma, \sigma)_i - \epsilon^{(k)}_i \big\|^2, $$

where $i$ indexes spatial positions. Under the denoising score-matching identity this error tracks the local mismatch between the learned score and the true score, which motivates reading it as a density signal.

The open-vocabulary known-class term is

$$ a_i = \log\sum_{c \in \mathcal{C}} e^{\langle f_i, t_c\rangle/\tau} - \log\sum_{c \in \mathcal{V}} e^{\langle f_i, t_c\rangle/\tau}, $$

where $t_c$ are text embeddings and $\tau$ is a temperature. $\mathcal{C}$ is the 19 Cityscapes class names. $\mathcal{V}$ is a broad vocabulary: $\mathcal{C}$ plus a large generic noun list such as the ADE20K-847 names. This is DenseHybrid's log-ratio idea with the fixed softmax replaced by a text-defined class set, so a pixel that matches "dog" better than any driving class scores as unknown.

The three per-pixel signals, $e_i(\sigma_\ell)$, $e_i(\sigma_h)$ and $a_i$, are calibrated with a Gaussian Neyman-Pearson test fitted on held-out normal Cityscapes validation pixels, following GROOD. A SAN-style high-resolution side branch, trained only on normal data, then upsamples the calibrated map along image edges. After detection, each flagged connected region is assigned its nearest vocabulary word, so the output is both a mask and a name.

Baselines

The naive reference is maximum softmax probability and softmax entropy of a closed-set Cityscapes segmenter. Fishyscapes documents that such models are catastrophically overconfident on road anomalies. This baseline shows how much any method adds.

The strongest existing method is DenseHybrid. It reports 78.0 AP on SMIYC AnomalyTrack and 0.2% FPR95 on SMIYC ObstacleTrack, adding 0.1 GFLOPs and under 3 ms on 2-megapixel images. We will run the released code under two conditions. The first is its published training setup. The second is a matched setup in which its heads sit on the same frozen vision-language features we use. The matched version separates backbone quality from scoring method. DenseHybrid trains with auxiliary negative data, so we will report our method both without negatives and with the same negatives added to the known-class term. Beating DenseHybrid means exceeding 78.0 AP on AnomalyTrack in the matched-data condition while not exceeding 0.2% FPR95 on ObstacleTrack. Matching it while adding the ability to name anomalies would be a weaker but still useful result.

The open-vocabulary competitor is zPROD's prompt-differencing pipeline (zPROD). It detects all foreground instances with a grounding vision-language model, then removes the instances matching known classes. It is reported to outperform supervised methods on RoadAnomaly21 and RoadObstacle21, but no reported result found on Lacuna for its exact AP or sIoU values. We will therefore rerun it with its ego-track filter on and off and report its latency alongside ours.

The key ablations isolate each ingredient. The receptive-field ablation keeps parameters and FLOPs fixed and swaps the context operator: 3×3 convolution stack, 7×7 windowed attention, criss-cross attention, and full attention, which is still feasible on an 8,000-token grid. The noise-level ablation reports $e(\sigma_\ell)$ alone, $e(\sigma_h)$ alone, and both. A text-free variant drops $a_i$. A heuristic-fusion variant replaces the Neyman-Pearson calibration with a tuned weighted sum. Every variant gets the same tuning budget: the same number of validation sweeps on Fishyscapes Lost & Found validation, with test tracks untouched.

Why not just…?

Why not just fit a density to per-location frozen features, with kNN or a Gaussian per pixel, and skip the diffusion model? This is a real threat. It is essentially GROOD's nearest-mean distance applied densely, and memory-bank distances are strong on industrial defects. It is also context-free by construction, which is exactly the regime our receptive-field hypothesis says should fail on in-context anomalies, such as a car-textured object where no car can be. We include it as a baseline. If it matches our criss-cross denoiser on SMIYC, the context claim is refuted and the contribution shrinks to the open-vocabulary known-class term.

Why not just train a masked feature predictor that predicts each location from its surroundings? At high noise our denoiser behaves much like one, so this is the most direct challenge to using diffusion at all. The difference is the low-noise term, which captures local density from the same network. We will train a masked predictor of matched size with criss-cross attention. If the masked predictor plus a separate local density model matches our two-level score, we will report that diffusion adds nothing beyond sharing one network.

Why not just run DiffuGTS-style inpainting with a few-step distilled sampler? Distillation reduces steps but keeps the inversion-and-inpainting loop. The file notes that low-step approximations lose fine textures and flag normal structure. We will run a four-step inpainting variant in our own feature space and compare AP against wall-clock time. The prediction is that it costs several times more for no gain in AP.

Why not just put DenseHybrid on a stronger backbone? The matched DenseHybrid baseline answers this directly. Even if it wins on AP, it cannot name the anomalies it flags. We would then report the method's value as naming at equal AP, not as detection.

Proposed Experiments

All training uses Cityscapes normal frames. Evaluation covers SMIYC RoadAnomaly21 and RoadObstacle21 (SegmentMeIfYouCan) with pixel AP, FPR95 and component sIoU, plus Fishyscapes Lost & Found and Static. We also report GFLOPs and milliseconds per 2-megapixel frame against DenseHybrid's under-3 ms overhead.

The decisive check is the receptive-field curve. AP should increase monotonically from the convolution denoiser to the criss-cross denoiser, and the gain should appear in $e(\sigma_h)$ but not in $e(\sigma_\ell)$. We will also split SMIYC anomalies by manually annotated "locally unusual" versus "contextually misplaced." The context gain should concentrate on the misplaced subset.

A second check targets naming. Using the mask-wise bipartite matching protocol of Revisit the Open Nature of Open Vocabulary Semantic Segmentation, we will report the front, back and err scores for anomaly names on a relabeled subset of RoadAnomaly21. Those scores are designed to tolerate plausible synonyms, which strict pixel matching would penalize.

A final check targets calibration. We will measure how far the realized false-negative rate on held-out normal frames deviates from the target rate set by the Neyman-Pearson test.

If the receptive-field curve is flat, the central claim is wrong, whatever the headline AP.