Tree-Shaped Feature Geometry for Real-Time Segmentation Errors at Boundaries

Fast segmentation networks label every pixel of a video frame within a few milliseconds by working at low resolution and then upsampling. At object boundaries and on thin objects, that upsampling blends the features of neighbouring regions. The blended feature lands somewhere between two class prototypes, and the network has to give it some label. This proposal starts from a geometric observation: those blended pixels behave like the "ghost samples" that continuous generative networks produce in the gaps between data modes. If the gap between "pedestrian" and "road" happens to lie nearest "pole", the pixel is called a pole. Current training puts every class at the same distance from every other class, so the label a blended pixel receives is close to arbitrary. Nothing stops it from being taxonomically absurd.

We propose a training-only regularizer that fixes where the class prototypes sit in feature space. Classes that share a parent in a label tree (for example "car" and "truck" under "vehicle") are placed close together, and distance from the origin encodes how specific a label is. Under this layout, a blend of two sibling classes stays inside the parent's region. A blend of two unrelated classes shrinks toward the origin, which the readout interprets as "commit only to a coarse category". At inference, the network walks down the tree for each pixel and stops at the finest node it can support. Split conformal prediction calibrates where it stops, and the walk costs a few comparisons per pixel. The backbone is unchanged, so frame rate is preserved. If this works, a real-time segmenter's inevitable boundary errors become "vehicle, unsure which kind" instead of "pole".

Core Claim

The claim has two parts that can be tested separately. The first is mechanistic. In real-time segmenters, a large share of severe mistakes, meaning mistakes that cross top-level categories, occur in a thin band around boundaries. These mistakes come from interpolated features falling into inter-class gaps, and the arrangement of the class prototypes decides which label the gap maps to. The second part is the method. Replacing the equidistant prototype arrangement with one shaped by the label tree reduces the severity of those mistakes and enables calibrated per-pixel back-off to parent categories. Fine-class mIoU should be unchanged or better, and inference cost should not increase.

The theoretical motivation comes from Mode Collapse and Regularity of Optimal Transportation Maps. That paper shows that when a target distribution has disconnected support, the optimal transport map onto it must be discontinuous. Continuous networks cannot represent this tearing, so they drop modes or emit ghosts in the gaps. The authors' fix (AE-OT) keeps the learned object continuous, a convex potential, and pushes the discontinuity into the gradient of that potential. We transfer that design principle rather than their algorithm. The segmentation network outputs a continuous feature. The discontinuity lives entirely in a cheap, exact readout that selects among nested cones. The feature geometry is arranged so that the continuous blur the network cannot avoid falls in places where it does the least damage.

Relation to Prior Work

Three lines of work motivate the gap. Understanding Imbalanced Semantic Segmentation Through Neural Collapse shows that segmentation features fail to settle into the ideal symmetric arrangement, and that rare classes get swallowed. Its Center Collapse Regularizer (CeCo) pulls class centers toward a simplex equiangular tight frame, an arrangement where every pair of class centers has the same cosine similarity $-1/(K-1)$ for $K$ classes. It does this with a training-only branch and adds no inference cost. We reuse the zero-cost auxiliary-branch design, but we reject its target geometry, because equal spacing is exactly the property that makes gap labels arbitrary.

Making Better Mistakes shows that lower top-1 error has not reduced how semantically severe mistakes are. It proposes hierarchical cross-entropy and tree-distance soft labels, but only for whole-image classification. Its soft labels also carry a cost: on iNaturalist-H, mistake severity improved from 2.41 to 2.10 while top-1 error rose from 43.77% to 55.16%. That result constrains our design. We want hierarchy to shape the geometry without flattening the fine-class targets.

Hyperbolic and nested-region embeddings supply the "radius encodes depth" idea. Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry places the root near the origin, and Imposing Category Trees Onto Word-Embeddings Using A Geometric Construction represents categories as nested balls. Neither was designed for dense, latency-bound prediction. We use a Euclidean version so that the readout stays a handful of dot products.

The multivariate-ordering literature explains why radius is the right readout. Monge–Kantorovich depth, quantiles, ranks and signs defines a canonical center-outward rank in $\mathbb{R}^d$ by transporting the data to a uniform ball, where the norm of the transported point is its rank and its direction is its sign. Neural Optimal Transport Meets Multivariate Conformal Prediction learns such maps with input-convex networks (networks constrained to be convex in their input) and turns them into conformal regions, but only up to $d=16$ and on tabular targets. Per-pixel input-convex transport over segmentation features is too expensive for real time. So we train the backbone so that its own feature norm and direction play the roles of rank and sign. The ordering question is then settled by construction rather than estimated afterwards. We test whether this holds by comparing against a post-hoc input-convex rank, as described below.

On uncertainty, Conformal Prediction in Hierarchical Classification with Constrained Representation Complexity gives coverage-guaranteed prediction sets on trees. Its search becomes exponential in the number of internal nodes allowed to span a set. We restrict every pixel's set to a single tree node, which makes the search a root-to-leaf walk. That restriction is what makes the approach feasible at video rate.

Research Program

The reference geometry is defined by a target inner product between class prototypes $\mu_i, \mu_j$ for leaf classes $i, j$:

$$\langle \mu_i, \mu_j \rangle = s_{h(i,j)}, \qquad s_0 < s_1 < \dots < s_{L-1},$$

Here $h(i,j)$ is the depth of the two classes' lowest common ancestor in an $L$-level tree, and $s_h$ is a fixed similarity that increases with shared depth. CeCo is the special case where $s_h$ is constant. We realise the targets with a nested construction. Each internal node gets a direction. Its children's directions are tilted from the parent's by a level-specific angle, and the children are spread as a small symmetric simplex in the subspace orthogonal to the parent direction. Leaf prototypes have unit norm and internal nodes have smaller norms. As in CeCo, a training-only head pulls batch class-mean features toward these fixed targets and is discarded at inference. The tilt angles and norms are the main hyperparameters, and we will sweep them on a small grid.

The readout walks the tree once per pixel. Starting at the root, it moves to the child whose direction has the highest cosine with the pixel feature. It stops when the feature norm falls below a level threshold $\tau_\ell$ or the child margin falls below $m_\ell$. The thresholds come from split conformal calibration on held-out pixels. Pixel correlation within an image is handled by calibrating per image and reporting image-level coverage, so that guarantees are not claimed under false independence. The target is that the returned node contains the true leaf for at least 90% of pixels.

The diagnostic program has three parts. The first, a boundary-band analysis, measures how mistakes are distributed as a function of distance to the nearest ground-truth boundary. Its purpose is to establish that severe mistakes concentrate where upsampling blends features. The second is a direct ghost test. We take real features from two adjacent regions, form convex combinations of them, and record which label each geometry assigns along the path. Flat prototypes should produce third-class labels in the middle, while tree geometry should produce the parent or the root. The third is a rank-consistency check. We fit an input-convex transport map, as in Neural Optimal Transport Meets Multivariate Conformal Prediction, on a 16-dimensional projection of calibration features and measure the Spearman correlation between its center-outward rank and our raw feature norm. High correlation would mean the network has learned to be its own ordering map. Low correlation would mean the post-hoc map should replace the norm in the readout, and we would report its cost.

Baselines

The naive reference point is STDC1-Seg50 from Rethinking BiSeNet for Real-Time Semantic Segmentation, trained with flat cross-entropy plus its detail loss. It reports 71.9% mIoU at 250.4 FPS on a GTX 1080Ti on Cityscapes. We will retrain it ourselves under our schedule so that all variants share data, iteration count, augmentation and hardware. Hierarchical back-off for this model is obtained by summing leaf softmax scores into parent nodes. Beating it means lower mistake severity and higher specificity at matched coverage, with fine mIoU within run-to-run noise or better.

The strongest competitor on the geometry question is STDC with CeCo. It is the only zero-latency feature-geometry regularizer in the file and differs from our method only in the target arrangement, which makes it the cleanest test of our claim. No reported result was found on Lacuna for CeCo on Cityscapes or with an STDC backbone. CeCo's reported gains are +1.6% on ADE20K with ResNet-101 and +1.8% on COCO-Stuff with Swin-T. We will run it with the same auxiliary-head capacity and the same tuning budget, a grid over loss weight equal in size to our grid over tilt angles. Beating it means matching its fine mIoU while lowering cross-category mistakes in the boundary band.

The strongest competitor on the hierarchy question is per-pixel hierarchical cross-entropy and soft labels from Making Better Mistakes, adapted to segmentation. Each is tuned over its decay or temperature parameter with the same budget. No reported result was found on Lacuna for these losses on dense segmentation. To beat them, we must reach their severity reduction without the fine-accuracy loss they show in classification.

Two ablations isolate the key ingredient. The first applies the tree targets with the flat readout (argmax over leaves), which separates the effect of the geometry from the effect of back-off. The second uses flat CeCo targets with our tree-walk readout, which shows whether calibrated back-off alone explains the gains. As a context point, Augmentation-Free Dense Contrastive Knowledge Distillation reports 77.03% mIoU for a ResNet-18 student on Cityscapes. That is a different, distillation-based regime and is complementary to ours. We will test whether adding our regularizer on top of it preserves its gain.

Why not just…?

Why not just use per-pixel hierarchical cross-entropy or soft labels? This is the most serious alternative. These losses change the output probabilities, but they do not control where blended features land relative to unrelated classes. Their classification results show a real accuracy cost (top-1 error 43.77% to 55.16% for soft labels on iNaturalist-H). The ablation grid settles the question directly. If per-pixel hierarchical cross-entropy matches our boundary-band severity and back-off specificity at equal fine mIoU, the geometric claim fails, and we will say so.

Why not just apply CeCo, which already fixes tail collapse at zero cost? CeCo fixes tail collapse by making every class equally distant from every other. Our ghost test is designed to show that this is exactly what makes boundary labels arbitrary. It is still possible that CeCo's tail-class gains outweigh our severity gains in practice. Reporting tail-class IoU alongside severity, and running the combination of CeCo-style balancing with tree targets, will show whether the two effects trade off or add.

Why not just calibrate back-off on flat softmax scores, with no change to training? Summing leaf probabilities into parents gives a hierarchical prediction set for free, and conformal calibration then guarantees coverage. The weakness is specificity. If flat training scatters a boundary pixel's probability across distant branches, the walk falls back to the root, so coverage is met but the answer is uninformative. The decisive measurement is average returned depth at 90% coverage for flat versus tree-shaped training. If they match, the training change adds nothing.

Why not just use a higher-resolution decoder? Higher-resolution decoding reduces blending, but it costs exactly the frame rate that defines the regime. The STDC variants already show that trade, from 71.9% mIoU at 250.4 FPS to 76.8% at 97.0 FPS. We will report severity for the higher-resolution variant too. The question is whether our method recovers part of that severity gain at the faster operating point.

Proposed Experiments

The main benchmark is Cityscapes, using its official two-level grouping of 19 classes into 7 categories. The validation split serves for calibration and testing, with images split disjointly. CamVid serves as a second driving set for latency and transfer checks. To test deeper trees, we use ADE20K (150 classes) and COCO-Stuff-164K. For both we build a WordNet-derived hierarchy, published with the code so that the hierarchy is not tuned to the method. All latency numbers come from the same GPU at a fixed input size.

Every variant reports the same metrics: fine-class mIoU, category-level mIoU, frames per second, and the average tree distance of mistaken pixels, overall and within a boundary band a few pixels wide. We also report recall on small and thin classes, and a dropped-class diagnostic. Following the logic of Seeing What a GAN Cannot Generate, which compares label histograms, this diagnostic compares the predicted per-class pixel-frequency histogram to the ground-truth histogram to detect classes the segmenter has effectively stopped producing. For the readout, we report empirical image-level coverage at a nominal 90% and the average depth of the returned node.

Three results would be decisive. First, the boundary-band analysis must confirm that severe mistakes in the flat baseline concentrate near boundaries. If they do not, the mechanism is wrong and the method has no reason to work. Second, in the ghost test, tree targets must convert third-class labels at feature midpoints into parent or root labels far more often than CeCo does. Third, at equal fine mIoU and equal frame rate, our method must lower cross-category mistakes in the boundary band and return deeper nodes at matched coverage than flat-softmax back-off does. The rank-consistency check then decides whether the cheap norm readout is sufficient or whether the post-hoc input-convex rank has to replace it. In the second case, we will report the latency cost honestly.