Stereo cameras on a vehicle can measure how far each pixel is, but a flat road does not look flat in that measurement. Near pixels have large disparity (the left–right pixel shift that encodes depth) and far pixels have small disparity. A pothole is only visible as a deviation from what the road should look like there. The strongest existing stereo pothole detectors therefore fit one smooth surface to the whole road, usually a parabola corrected for camera roll, and subtract it. This works on straight, crowned-free roads. It breaks on banked turns, multi-lane crowns, speed bumps and intersections, because one surface cannot describe them. The cost is real: road inspection is still largely manual, slow and dangerous. Automated systems that fire false alarms on every curve will not replace a human surveyor.
We propose to replace the hand-fitted road surface with a learned one. A network predicts the expected undamaged road disparity at each location using only distant parts of the image, never the pixel's own neighborhood. Because the prediction cannot see the local patch, it cannot "explain away" a pothole, yet it can follow banking, crowns and slope changes that are visible elsewhere in the scene. The leftover residual, measured minus predicted, becomes a clean geometric anomaly signal. That signal is then fused with the color image through a cross-attention gate driven by stereo-matching confidence, so color takes over where the disparity is unreliable (wet asphalt, glare, low texture). Long-range attention is not a capacity add-on here. It is what makes the reference surface both flexible and blind to the defect.
Core Claim
Disparity normalization can be recast as masked long-range regression: predict road disparity at pixel $p$ from context outside a radius $r$ around $p$. This yields a reference surface that matches parametric normalization on simple roads and clearly beats it on non-planar geometry, without the false gradients a single-surface fit produces. Concretely, a network $f_\theta$ outputs $\hat d_{\text{ref}}(p)$ from disparity and color features at locations $q$ with $\|q-p\| > r$. It is trained on presumed-road pixels $R$ with a robust, confidence-weighted loss
$$\mathcal{L} = \sum_{p \in R} c(p)\,\big|\,d(p) - \hat d_{\text{ref}}(p)\,\big|,$$
where $d(p)$ is the measured stereo disparity and $c(p)\in[0,1]$ is a stereo-matching confidence, for example from left–right consistency of Semi-Global Matching (SGM). Potholes are a small fraction of road pixels, and the L1 loss is robust to outliers. The model should therefore learn the undamaged surface even though no defect labels are used in this stage. The anomaly signal is the residual $\Delta d(p) = d(p) - \hat d_{\text{ref}}(p)$.
What Exists and What Is Missing
Classical normalization supplies the problem definition and the main geometric baseline. Two approaches project disparity into spaces where the road becomes a line or a plane. Identifying Good Training Data for Self-Supervised Free Space Estimation harvests ground labels from the v-disparity curve, a histogram of disparity per image row. Obstacle detection by ground homograph estimation during autonomous navigation flags pixels that violate a single ground homography. Fan and colleagues refined this into parametric disparity transformation. They fit a parabola in A Novel Disparity Transformation Algorithm for Road Segmentation, estimate roll in Pothole Detection Based on Disparity Transformation and Road Surface Modeling, and solve for tilt in closed form in Road Damage Detection Based on Unsupervised Disparity Map Segmentation. The most complete version, Rethinking Road Surface 3-D Reconstruction and Pothole Detection, adds a generalized perspective transformation, superpixels and k-means thresholding. PT-ResNet feeds a perspective-transformed input to a residual network. All of these share one global surface model, and that is the limitation we target. We reuse their v-disparity harvesting to define $R$, and we keep their parametric model as a baseline and as an optional input feature.
The long-range attention work supplies the mechanism and motivates the masking design. Attention lets distant pixels interact in one step. It restored continuity of broken structures in Advantage of Machine Learning over Maximum Likelihood in Limited-Angle Low-Photon X-Ray Tomography. It linked pixels across a warped geometry in SynDistNet. It resolved low-texture ambiguity in Attention Aware Cost Volume Pyramid Based Multi-view Stereo Network. UDTIRI reports that Transformer models beat convolutional ones on small potholes but are slow on edge hardware. That latency finding constrains our design. Since the road reference is smooth, we predict it on a coarse token grid, about 1/16 resolution, where quadratic attention is cheap. We upsample only the reference, not the full segmentation network.
Cross-modal attention supplies the fusion step. Depth-completion work, Gated Cross-Attention Network for Depth Completion, lets sparse depth and color guide each other. An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention gates cross-modal context against native features to stop one modality from dominating. JL-DCF documents that naive RGB-depth fusion lets color swamp geometry. We adopt the gate but tie it to stereo confidence $c$ rather than learning it freely.
A caveat on novelty: the research file's section on closest existing work was missing, and its cross-modal open-problems section is truncated. "RGB plus transformed disparity with attention fusion" is therefore the obvious combination, and it may already be published. The claim we stake is narrower and, as far as the file shows, unaddressed: the masked, distance-restricted learned reference surface, evaluated specifically on non-planar road geometry.
Research Program
The system has three stages.
Stage 1: data and road mask. We compute SGM disparity and left–right confidence for each stereo pair. We obtain a presumed-road mask from v-disparity harvesting, with no human labels.
Stage 2: reference network. This is a small hybrid model. A convolutional stem feeds a coarse-grid Transformer whose attention is masked so that each query token ignores keys within radius $r$. It is trained with the loss above on unlabeled driving stereo. The radius $r$ is the central knob. Too small and the model can copy the pothole into the reference. Too large and it loses the local slope information needed for speed bumps. We will sweep $r$ against typical pothole diameter in pixels at each image row. Because apparent size shrinks with distance, $r$ may itself need to scale with row; testing a row-dependent radius is part of the program.
Stage 3: segmentation head. The head takes color features as queries and residual-disparity features as keys and values. The output is $g\cdot\text{Context} + (1-g)\cdot\text{RGB}$, with the gate $g$ computed from $c$, so geometry is trusted only where stereo is reliable. This head is trained on the labeled pothole data.
The analysis component asks whether the residual is cleaner. We measure the residual's variance on undamaged road, split by road-geometry category. We also measure the separation between pothole and road residuals using area under the ROC curve on pixels. Both are computed before any segmentation head is trained, so Stage 2 can be judged on its own.
Baselines
The naive reference is raw disparity with no normalization, concatenated with RGB and fed to the same segmentation network. This shows how much any normalization helps.
The strongest geometric baseline is the pipeline of Rethinking Road Surface 3-D Reconstruction and Pothole Detection. It reports an overall F-score of 89.42%, 99.6% pixel accuracy, and 3.5 ms for transformation plus detection on an RTX 2080 Ti, on its own stereo pothole data. We will rerun it on identical SGM disparity. We will report it unsupervised, as published, and also with its transformed disparity fed into our segmentation head, so the only difference from our method is the reference surface. Beating it means at least matching 89.42% F-score on its dataset and clearly exceeding it on the non-planar split.
The strongest learned baseline is an RGB-only Transformer segmenter of the kind UDTIRI finds best for small potholes. It gets the same backbone, training schedule and tuning budget. No reported result was found on Lacuna for UDTIRI's per-model scores, so we will rerun it ourselves rather than quote a number.
Two ablations isolate the key ingredient:
- Unmasked reference ($r=0$), the same network but allowed to see the local patch. This tests whether masking matters.
- Local-only reference, with attention restricted to within $r$. This tests whether long range matters.
A further ablation swaps the confidence gate for plain concatenation and for an ungated cross-attention, to isolate the fusion choice.
All learned methods share data splits, disparity inputs and a fixed number of tuning trials. Runtime is measured on one desktop GPU and one embedded GPU.
Why not just…?
Why not just give raw disparity and RGB to a strong Transformer segmenter and let it learn normalization implicitly? This is the most likely competitor, and it may close much of the gap when labeled data are plentiful. However, pothole labels are scarce: the file stresses data scarcity in inspection. Implicit normalization must also be learned from those few labels, whereas our reference is learned from unlabeled stereo. The naive-reference baseline answers this directly. We will repeat it at 10%, 30% and 100% of labels. If the advantage disappears at full labels and persists only at low labels, we will report the method as a label-efficiency result, not an accuracy result.
Why not just fit piecewise or patch-wise parabolas with robust fitting (RANSAC) instead of one global surface? This is a serious threat because it is cheap and directly targets the single-surface failure. Its weakness is the same one our masking avoids. A patch small enough to follow a banked curve can be dominated by a large pothole inside it, which fits the defect into the reference and erases it. We will include a tuned patch-wise RANSAC parabola baseline and compare residual separation as a function of pothole size relative to patch size. If it matches us across sizes on the non-planar split, the learned reference is unnecessary and we will say so.
Why not just improve stereo matching, since low-texture noise is a dominant error source? Better disparity helps every method equally but does not fix the wrong surface model on banked or crowned roads. We will test this by rerunning the main comparison with a stronger learned stereo matcher in place of SGM. Normalization error, our target, should stay distinct from matching error.
Proposed Experiments
Benchmarks. We use the stereo pothole data released with the Fan et al. disparity-transformation papers, for direct comparability with the 89.42% F-score. We use UDTIRI for the RGB-side comparison and small-pothole evaluation. If UDTIRI lacks stereo pairs, it is used only for the color baseline and the fused model's color branch.
The decisive test is a non-planar geometry split. We will collect and annotate a modest set of stereo sequences covering banked turns, crowned multi-lane roads, speed bumps and intersections. We will complement it with a controlled stress test: apply known smooth non-planar warps (banking angle, crown height) to disparity maps of flat-road scenes with labeled potholes. This lets us plot false-positive rate on undamaged road against curvature magnitude for each method.
Metrics. Pixel F-score and intersection-over-union for potholes, false positives per frame on undamaged road, the residual-separation curve above, and latency.
Decisive checks.
- The parametric baseline's false positives should grow with curvature while ours stay roughly flat.
- The masked reference should beat both the unmasked and the local-only ablations on large potholes.
- On flat-road data, we should not lose more than about one F-score point to the parametric pipeline.
- The coarse-grid reference should add only a few milliseconds over the 3.5 ms classical transformation.
If the second check fails, the claim that long-range masking is the essential ingredient is wrong.
References
- Identifying Good Training Data for Self-Supervised Free Space Estimation
- Obstacle detection by ground homograph estimation during autonomous navigation
- A novel disparity transformation algorithm for road segmentation
- Pothole Detection Based on Disparity Transformation and Road Surface Modeling
- Road Damage Detection Based on Unsupervised Disparity Map Segmentation
- Rethinking Road Surface 3-D Reconstruction and Pothole Detection: From Perspective Transformation to Disparity Map Segmentation
- PT-ResNet: Perspective Transformation-Based Residual Network for Semantic Road Image Segmentation
- Advantage of Machine Learning over Maximum Likelihood in Limited-Angle Low-Photon X-Ray Tomography
- SynDistNet: Self-Supervised Monocular Fisheye Camera Distance Estimation Synergized with Semantic Segmentation for Autonomous Driving
- Attention aware cost volume pyramid based multi-view stereo network for 3D reconstruction
- UDTIRI: An Open-Source Road Pothole Detection Benchmark Suite
- Gated Cross-Attention Network for Depth Completion
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection