Most photos are taken by people standing upright and aiming roughly level. Datasets therefore over-represent eye-level, horizontal views. A robot or drone that looks down at the floor sees the same objects at very different apparent sizes and with strong foreshortening, meaning parts nearer the camera look stretched and parts farther away look squashed. Segmentation models collapse in this regime. On a Matterport3D benchmark where the camera tilts from level to straight down, one study reports 1.53 mean intersection-over-union (mIoU) for an unadapted model at the steepest tilt (ADeLA). A separate line of work builds networks that keep a full-resolution feature map alive through the whole network depth instead of shrinking it and upsampling at the end (HRNet). These networks are good at small and thin structures. Nobody has asked what they do when the camera tilts.
This note proposes that a large share of tilt-induced failure is a scale problem in disguise. Tilting the camera changes the apparent size of objects differently at each pixel. A network trained on level views has learned, implicitly, which resolution stream to trust at each image location for the size distribution it saw. We propose to give a multi-resolution network a cheap, closed-form map of how much each pixel's apparent scale has moved away from the canonical view. That map comes from camera pitch and height, which an onboard inertial sensor provides. The map then decides, pixel by pixel, how the high- and low-resolution streams are mixed. The work starts with a diagnostic study that can falsify this premise before any method is built.
Core Claim
Canonical viewpoint bias, as profiled in web and academic datasets (Qiu et al.), is partly a bias in the joint distribution of image location and apparent object scale. High-resolution networks fuse their streams with weights that do not depend on viewpoint. In HRNet these are repeated cross-scale exchanges. In Lite-HRNet they are conditional channel weighting units, which reweight channels using pooled statistics from all resolutions. Either way, the fusion encodes that canonical scale layout.
We claim two things. First, conditioning the fusion per pixel on an analytic "scale displacement" map should recover a meaningful part of the lost accuracy at moderate-to-steep pitch, roughly 20° to 60°, with no target-domain images and negligible extra compute. Second, the same conditioning applied globally, or through raw coordinate channels, should not recover it. The second claim is what makes the mechanism specific. The claim does not extend to near-90° views, where the visible content itself changes (the horizon disappears and the floor dominates). Scale reasoning cannot fix that, and we treat it as an honest limit.
The Mechanism
For a camera at height $h$ above a ground plane, pitched down by angle $\theta$, consider a pixel whose ray lies an angle $\alpha$ below the optical axis. For a pinhole camera with focal length $f$ and principal row $c_y$, $\alpha = \arctan((v - c_y)/f)$ for image row $v$. The ray reaches the ground at slant range $r = h / \sin(\theta + \alpha)$. A fixed-size object there appears with scale proportional to $f/r$. We define the log scale displacement at pixel $(u,v)$ as
$$ \delta(u,v) = \log \frac{\sin(\theta + \alpha)}{\sin(\theta_0 + \alpha)}, $$
where $\theta_0$ is the canonical training pitch, here 0°. $\delta$ says how much larger or smaller things at that pixel look than they did, at the same pixel, during training. For rays at or above the horizon, the ground-plane formula is undefined. There we fall back to a learned constant. We also test a depth-derived version of $\delta$, built from Habitat ground-truth depth, as an oracle upper bound.
In each fusion block, the contribution of resolution stream $k$ at a pixel is multiplied by a gate $g_k(u,v) = \mathrm{softmax}_k\, \phi(\delta(u,v))$. Here $\phi$ is a tiny shared function: a two-layer perceptron over $\delta$ and its local gradient. In the Lite-HRNet variant, the same gate modulates the spatial weights inside each conditional channel weighting unit. The added cost is one per-pixel scalar map and a few hundred parameters.
This differs from three neighbouring approaches. Continuous Viewpoint Adaptation morphs all convolution weights globally as a function of camera angles via a neural ordinary differential equation, and is evaluated only on isolated CAD objects. Our conditioning is spatially varying, closed-form, and applied to cluttered scenes. FOCAL canonicalizes the input image at test time with diffusion and CLIP search. That is slow, and generative re-rendering can alter fine edges, whereas we change nothing in the image. ADeLA transports labels across views using unlabeled target images. We use no target images at all, so the two are complementary.
Research Program
The first stage is a diagnostic study, and it decides whether the method is worth building. On the Habitat/Matterport3D pitch benchmark from ADeLA, we train four segmentation backbones on 0° views only and evaluate across all target pitches. Two are high-resolution: HRNetV2-W48 and Lite-HRNet-30. One is the single-scale transformer UViT, which keeps a constant 1/8 feature scale. The fourth is a downsample-then-upsample baseline. Habitat renders depth, so we can bin every test pixel by its true scale displacement. That lets us split each model's error into two parts. One part comes from scale displacement: classes that are present in training views but appear at a different size. The other comes from content novelty: regions such as top-down floor that have no level-view counterpart.
The premise survives only if two things hold at moderate pitch. A substantial fraction of error must sit in high-$|\delta|$ pixels of seen classes. And high-resolution networks must differ from the downsampling baseline specifically along the $\delta$ axis. If instead error is dominated by content novelty, the gating idea is wrong for this problem. In that case we report the decomposition as the finding. It would still answer an open question: whether keeping full-resolution features makes networks more or less fragile to tilt.
The second stage builds the scale-gated fusion into HRNet and Lite-HRNet. Training uses 0° data plus synthetic tilt: rotation homographies applied to 0° images, with the corresponding $\delta$ map. Rotation about the optical centre is exactly a homography, so these warps are geometrically correct in the region they cover.
The third stage tests real data. Objectron videos carry ARKit/ARCore camera poses with gravity. The Objectron authors note that standing-height capture biases elevation coverage. We split its frames by camera elevation and test whether the ordering of methods from Habitat holds on real handheld captures.
Baselines
The naive reference point is each backbone trained on 0° views and tested unchanged. For HRNet, an implementation sanity check is required: before any viewpoint experiment, our HRNetV2-W48 must reproduce close to the 81.1% Cityscapes mIoU reported in HRNet.
The strongest published method on this benchmark is ADeLA. It reports 67.63 mIoU at 20° pitch and 5.03 mIoU at 90°, against 1.53 unadapted and 2.41 for the best prior unsupervised domain adaptation method. It is the strongest competitor because it is the only method in the file built for exactly this pitch shift on dense segmentation. ADeLA uses unlabeled target-pitch images and we do not, so we run it on our backbones with its published protocol and report it as a stronger-information reference. We also run ADeLA combined with our gating, to test complementarity. Beating ADeLA alone at 20° to 60° without target images would be a strong result. Matching it would still be meaningful given the information gap.
The most dangerous practical baseline is homography rectification. Given known pitch, the tilted image is warped back to a virtual 0° view before running the unchanged high-resolution network. A second baseline trains with the same synthetic tilt augmentation we use, but without gating.
The ablations that isolate our ingredient keep that augmentation fixed and vary only the conditioning. The variants are global pitch conditioning via feature-wise affine modulation, pitch plus pixel-coordinate input channels, our analytic $\delta$ gating, and oracle depth-based $\delta$ gating. All variants share data, schedule, and a matched hyperparameter search budget per method, with three seeds each. Our method counts as winning if it beats the augmentation-only and coordinate-channel variants at 30° to 60° by more than seed variance. Its gain should also concentrate in high-$|\delta|$ pixels, as the diagnostic predicts.
Why not just…?
Why not just rectify the image with a homography using known pitch? For small tilts this is exact and will likely match or beat us at 10° to 20°. Beyond about 40°, the rectified image either leaves most of the tilted view outside the virtual 0° frame or must stretch near-camera pixels many times over. The second option creates exactly the blur that high-resolution networks were built to avoid. Our pitch sweep reports both methods at every angle. If rectification wins everywhere, we will say so and drop the gating claim.
Why not just augment with synthetic tilts and let the network learn? This is a real threat, and it is why augmentation is held constant across our ablations rather than treated as an alternative. RoboView-Bias found that agents trained with broad randomization still swing sharply under small camera rotations. That suggests augmentation alone does not teach a network where to rely on fine versus coarse features. The augmentation-only versus gated comparison settles this directly.
Why not just feed pitch or pixel coordinates as extra inputs? In principle a network could compute $\delta$ itself from these. These variants are therefore our closest ablations. If they match analytic gating, the contribution shrinks to "give the network the camera pitch," and we would report that plainly.
Why not just scale up pretraining, as Sapiens does? Sapiens pretrains up to 2B-parameter transformers at native 1024-pixel resolution on 300M human images and notes qualitative robustness to unusual angles. It never isolates viewpoint, and its compute is unavailable for the edge regime where Lite-HRNet operates at 0.7 GFLOPs. We include a frozen large pretrained transformer with a dense head as a capacity reference, while acknowledging that it is not a compute-matched comparison.
Why not just use domain adaptation? ADeLA showed that feature-alignment methods cause negative adaptation under content shift. ADeLA itself is included as a baseline above.
Decisive Checks
Three outcomes determine whether this idea survives. First, the depth-binned decomposition must show that error at moderate pitch concentrates in scale-displaced pixels of seen classes. Second, analytic $\delta$ gating must beat both augmentation-only training and coordinate-channel conditioning at 30° to 60°, and come within reach of the oracle-depth gate. Third, the ordering of methods must hold on elevation-binned Objectron frames. If the first check fails, the paper becomes an empirical study of high-resolution networks under tilt. If the second fails, it becomes a negative result about explicit geometric conditioning. Either is still a contribution, because no prior work in the file has measured this interaction.
References
- ADeLA: Automatic Dense Labeling with Attention for Viewpoint Adaptation in Semantic Segmentation
- Deep High-Resolution Representation Learning for Visual Recognition
- Qualification and Quantification on Viewpoint Biases in Large Scale Image Datasets for General Object Recognition
- Lite-HRNet: A Lightweight High-Resolution Network
- Continuous Viewpoint Adaptation for Single View 3D Object Reconstruction
- Test-Time Canonicalization by Foundation Models for Robust Perception
- A Simple Single-Scale Vision Transformer for Object Detection and Instance Segmentation
- Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild With Pose Annotations
- RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation
- Sapiens: Foundation for Human Vision Models