Horizon-Anchored Coordinates for Driving-Scene Segmentation on Unseen Cameras

Standard convolutional networks apply the same filter at every pixel, so they cannot directly use where a pixel sits in the image. In driving scenes, position is a strong cue: sky sits above the horizon, road below it, and object size shrinks toward the vanishing point. CoordConv, which appends fixed $x,y$ coordinate maps to the feature maps, lets a network read position. But fixed coordinates describe pixel positions on the sensor, not positions in the scene. A model trained on one dashcam, at one height, pitch and focal length, learns that "road starts at row 0.55". On a different vehicle that rule is wrong. Domain generalization methods train on one set of domains and must work on unseen ones with no target data. They have focused almost entirely on photometric shift (texture, lighting, weather) and treat image layout as a stable anchor.

We propose to give the network coordinates measured relative to the scene rather than the sensor. A tiny head predicts the horizon row and a perspective scale from the image itself. Every coordinate channel is re-expressed in that frame. The head is trained without any camera labels, only by requiring its predictions to shift and scale consistently when we apply known crops and zooms during training. In two sentences: give the network position information, but in a camera-normalized frame it estimates itself, so the positional prior transfers across camera rigs. Then test, on synthetic-to-real segmentation and on controlled camera-shift stress tests, whether this beats both plain CoordConv and geometric augmentation alone.

Core Claim

Zero-shot domain generalization in structured scenes suffers from a second, mostly ignored shift: geometric shift from camera mounting and optics. Two simple responses each fail on one side. Fixed coordinate channels exploit layout but overfit to the source camera. Coordinate-free networks with heavy crop augmentation are robust to the camera but throw the layout prior away. Self-estimated, horizon-anchored coordinates should keep the prior and drop the camera dependence.

The prediction this makes is specific. The gain over a coordinate-free baseline should grow with the source–target geometric gap. The gain over fixed CoordConv should be largest exactly where fixed CoordConv degrades below the coordinate-free baseline.

Where this sits in prior work

An intriguing failing of convolutional neural networks and the CoordConv solution supplies the mechanism. It showed that convolutions without coordinates cannot learn coordinate transforms, and that CoordConv extrapolates to a held-out quadrant. Its experiments keep the visual domain fixed, and it reports essentially no ImageNet gain (+0.04% top-1). That is a constraint on us: coordinates help only where layout carries signal, which is why we target driving scenes rather than object classification.

Two lines of work motivate the camera-normalization step. GEDepth: Ground Embedding for Monocular Depth Estimation embeds a ground-plane depth computed from known camera intrinsics and extrinsics. For KITTI→DDAD transfer it reports Abs Rel falling from 0.644 to 0.261. This is the strongest evidence that geometry overfitting is real and fixable. However, it needs calibration at test time and is specific to depth. We need neither. Geometry-Aware Network for Domain Adaptive Semantic Segmentation lifts pixels to 3D and finds large gains on thin objects such as poles and traffic lights. It is domain adaptation, though, and needs unlabeled target images during training; our setting forbids them. Discovering Spatial Relationships by Transformers for Domain Generalization argues that spatial relations are domain-invariant and models them with attention over CNN tokens. It does not ask whether absolute position helps or hurts across domains, which is our question.

Photometric domain generalization supplies the base systems we build on and must beat. RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening suppresses style-sensitive feature covariances. Its authors name geometric and camera-specific variation as future work, which is precisely our target. Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation (SHADE) synthesizes new feature styles and enforces consistency. Pin the Memory: Learning to Generalize Semantic Segmentation meta-learns appearance-only class memories. All three are orthogonal to our module, so the method is designed to plug into them.

Guided Texture Segmentation via Coordinate-Aware Class-Ratio Mapping shows that a learned gate on coordinate features outperforms naive concatenation, but only in-domain. We borrow its gating idea. Deep Spatial Domain Generalization generalizes over geographic coordinates with graph hypernetworks. It shares the word "spatial" but operates on latitude and longitude, not image coordinates.

Research Program

The module has three parts.

Horizon head. A light head on an early backbone stage regresses two numbers: a horizon row $h$ and a log-scale $\log s$, where $s$ is a proxy for focal length.

Canonical coordinates. For a pixel at normalized row $y$ and column $x$, we define coordinates relative to an assumed central vanishing column. We also add a channel $r$, the inverse distance from the horizon, which approximates ground-plane depth up to scale:

$$ \tilde{y} = \frac{y - h}{s}, \qquad \tilde{x} = \frac{x - 0.5}{s}, \qquad r = \frac{1}{\max(\tilde{y}, \epsilon)} $$

Here $\epsilon$ is a small floor that keeps $r$ finite above the horizon. These channels replace fixed CoordConv channels at the decoder and in the last two encoder stages. Each injection passes through a sigmoid gate computed from appearance features, so the network can down-weight position where layout is uninformative.

Equivariance training. The horizon head receives no labels. During training each image is shown twice, under two random geometric transforms with known parameters: a vertical crop offset $\delta$, which simulates pitch, and a zoom factor $z$, which simulates focal length. Under these transforms a correct horizon and scale must transform predictably, giving the loss:

$$ \mathcal{L}_{eq} = \left| h' - z\,(h - \delta) \right| + \left| \log s' - \log s - \log z \right| $$

Here primes denote predictions on the transformed view. This consistency alone fixes the predictions only up to an offset shared by all images, so we add one weak anchor: a loss pulling $h$ toward the boundary between sky/building and road labels in the source images, which GTAV already provides. The anchor uses training labels only, so the zero-shot protocol is preserved.

Beyond the method, we will build a diagnostic. Cityscapes, BDD100K and Mapillary validation images will be re-rendered through controlled crop-and-resize transforms that simulate pitch changes of several degrees and focal-length changes of ±30%. This gives curves of accuracy against geometric gap, separate from photometric gap. We will also measure the empirical horizon distribution of each dataset from labels, so natural benchmarks can be ordered by geometric distance from GTAV.

Baselines

The naive reference is an ERM-trained (plain supervised training on the source) DeepLabV3+ ResNet-50 on GTAV with standard flip/crop augmentation. RobustNet reports 28.95% mIoU for this kind of baseline on GTAV→Cityscapes. We will re-run it in our pipeline rather than rely on that number.

The strongest same-regime competitors are SHADE and Pin the Memory. SHADE is run from its official code and reports an average of 42.42% mIoU over three real targets from GTAV alone. Pin the Memory reports 44.51% on Cityscapes, 38.07% on BDD100K and 42.70% on Mapillary from GTAV. We re-run both with the published recipes and identical backbones, schedules and crop sizes. Our module is added on top of each with no extra tuning beyond a sweep over the equivariance-loss weight of at most five values, chosen on a held-out GTAV split, never on targets. Beating them means a mean gain of at least 1.5 mIoU across Cityscapes, BDD100K, Mapillary and IDD, over three seeds, with non-overlapping seed ranges on at least three of the four targets.

RobustNet's 36.58% on GTAV→Cityscapes is a secondary reference. Foundation-model methods reach above 71% according to Domain Generalization for Semantic Segmentation: A Survey. They are a different compute regime and are addressed separately below.

The ablations that isolate the key ingredient each change one thing:

  • fixed CoordConv channels from the official implementation, with the same gate;
  • the same model with the full geometric augmentation used for $\mathcal{L}_{eq}$, but no coordinates;
  • our coordinates with the horizon head trained only on the label anchor, without equivariance;
  • an oracle that feeds the horizon computed from target labels, which upper-bounds what a better horizon estimator could give.

The second ablation is the decisive one.

Why not just…?

Why not just use aggressive geometric augmentation with a coordinate-free CNN? This is the real threat. Random crops and zooms might make the network robust to camera changes without any coordinates. The file offers no direct evidence against it in segmentation. The CoordConv quadrant results show coordinate-free convolutions struggle to compute position-dependent functions, but that is a synthetic task. Our "augmentation, no coordinates" ablation receives exactly the same geometric transforms as our method, so the only difference is the coordinate frame. If it matches us on natural targets and on the stress-test curves, we conclude that the layout prior is not worth recovering, and we report that as the finding.

Why not just append fixed CoordConv channels and augment? Augmenting fixed coordinates teaches the network that position is unreliable, which pushes the gate toward zero and discards the prior. Not augmenting lets the network overfit to the GTAV camera. We expect fixed coordinates to sit above the coordinate-free baseline at small geometric gap and to cross below it as the gap grows. The stress-test curves show this crossing directly. If fixed CoordConv never crosses below, the camera-normalization part of the claim is wrong.

Why not just use photometric methods like ISW or SHADE? They target a different shift. RobustNet's authors themselves flag geometric variation as uncovered. Because our module stacks on SHADE and Pin the Memory, the question is additive gain, not replacement. Under our hypothesis, a photometric method alone should show flat gains along the geometric stress axis, while our module should show gains that grow along it.

Why not just use a frozen foundation model with positional embeddings? Methods built on frozen vision-language backbones far exceed CNN numbers, at over 71% according to the survey. We do not claim to beat them. Their learned positional embeddings are still sensor-relative, so the same camera-overfitting question applies. As a secondary experiment, we inject horizon-anchored coordinates as an extra additive positional term in a ViT-based segmenter's adapter. We then test whether its stress-test curve flattens, judged on the slope of that curve rather than on absolute mIoU.

Proposed Experiments

The main experiment is GTAV→{Cityscapes, BDD100K, Mapillary, IDD}, plus GTAV+SYNTHIA→real, scored by mIoU over 19 classes with three seeds. We will report per-class results with attention to road, sidewalk, sky, pole and traffic sign, the classes most tied to layout.

The first decisive check is the geometric stress test. We plot mIoU against simulated pitch and focal offset for four models: ERM, augmentation without coordinates, fixed CoordConv, and ours. The hypothesis requires our curve to be the flattest while staying at or above the best model at zero offset.

The second check is horizon quality. Error between the predicted horizon and the label-derived target horizon should correlate with per-image gain. If gains appear even when the predicted horizon is wrong, the coordinates are not doing what we claim.

The third check is cross-task. We repeat the module for KITTI→DDAD monocular depth, uncalibrated, and compare against GEDepth's calibrated 0.261 Abs Rel. Approaching that number without camera parameters would show that horizon-anchored coordinates recover much of what explicit calibration gives. Falling far short would bound the method's reach.