Coaxial Sensors and the Boundaries RGB Cannot See

Segmentation networks fail most often at the edge of an object, and they fail worst where the object and its background look the same. A flat polyp has the same color as the colon wall. A pedestrian at night is nearly black against a black street. Most research attacks this in software, adding modules that guess harder at the fuzzy edge. A second sensor tuned to a different physical property, such as near-infrared reflectance, emitted heat, or distance, can show a real intensity step at exactly the pixels where the color camera shows none. That step is only useful if it lands on the same pixel. Side-by-side cameras see the scene from slightly different positions, a displacement called parallax, and warping one image onto the other smears the very boundary pixels in question. Coaxial rigs avoid this. They use a beam splitter or prism so that both sensors share one optical center, and every pixel lines up with no depth estimate needed.

This note proposes a controlled study with one targeted architectural change. We use coaxially aligned data and its accompanying depth to answer three questions. How much of a second modality's segmentation benefit is concentrated at low-contrast boundaries? How fast does that benefit disappear as we reintroduce physically simulated parallax of increasing baseline? Does routing encoder "skip connections" by which sensor has local contrast recover more of it than simply stacking channels? Skip connections are the shortcuts in a U-Net (an encoder-decoder segmentation network) that pass high-resolution encoder features straight to the decoder. The main deliverable is a tolerance curve: boundary accuracy against misregistration, stratified by how much contrast the color image has at each boundary pixel. No paper in this literature measures that curve, and it is what a hardware designer needs to decide whether a coaxial rig is worth its cost in light and weight.

Core Claim

We predict three things. First, the gain from the second modality lives almost entirely in boundary pixels where the RGB image has low local contrast. On aggregate area metrics like mIoU the gain looks modest; on boundary metrics in that stratum it is large. Second, the gain falls off sharply once misregistration exceeds the boundary-metric tolerance, a few pixels. We expect it to become negative near depth discontinuities, where parallax creates regions that one sensor sees and the other does not. Third, the network needs to choose which sensor's shallow features to trust at each boundary pixel. A U-Net whose skips are gated by per-pixel, per-modality local contrast should beat input concatenation in the low-RGB-contrast stratum. In that stratum, RGB-only boundary-refinement modules should add little once the auxiliary channel supplies a physical edge. Where neither sensor has contrast, they should still help.

Position Relative to Prior Work

The coaxial hardware literature supplies the data and the motivation. Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision builds dichroic-prism camera heads that give pixel-aligned RGB and 750–1000 nm NIR, together with LiDAR depth. It evaluates only YOLO detection (mAP 0.756 to 0.809 with fusion) and stereo depth, never dense segmentation. Multispectral pedestrian detection: Benchmark dataset and baseline uses a beam splitter to align RGB and thermal. With hand-crafted features it reduces the nighttime miss rate from 90.17% to 63.99%, but it too stops at bounding boxes. Instrument segmentation in hybrid 3-D endoscopy using multi-sensor super-resolution and ToF Meets RGB are the only works here that segment on coaxial data. They combine thresholded color and super-resolved range masks with a boolean AND and reach F-score 0.73. Their setup also shows the resolution mismatch problem, 640×480 RGB against 64×48 time-of-flight, that shapes our gating design. All four establish that coaxial data can be captured. None asks how much pixel-exact alignment matters at boundaries, and that is our question.

The boundary-ambiguity literature supplies the refinement techniques we test against, and it constrains our claims. Polyper separates a coarse mask into interior and "potential boundary" bands with fixed erosion and dilation, then refines the band with cross-attention. Removing that separation costs 4.40 mIoU. MISNet and the Res2Net reverse-attention network instead suppress confident foreground so that later stages focus on the fuzzy margin. MISNet's authors explicitly name multi-modal integration as future work. Niu et al. add a contour branch trained with a 0.9/0.1 segmentation/contour loss and cut the spleen 95% Hausdorff distance from 7.5744 to 5.8814. We reuse this loss unchanged. All of these work on a single modality. Where the sensor records no contrast, they can only impose priors, and our stratified evaluation is designed to show where that limit bites.

The RGB-thermal segmentation literature is the closest multi-modal competitor. Complementary Random Masking (CRM) masks complementary patches across RGB and thermal and adds self-distillation, reaching 61.2% mIoU on MFNet. FEANet fuses the two streams by attention-weighted summation, and MVNet reaches 40.48% nighttime mIoU on MVSeg. These methods use data aligned by software or unanalyzed hardware, and they report area metrics only. Our work differs in two ways. We treat alignment as a controlled variable, and we measure at boundaries, stratified by contrast.

Research Program

Data. Coaxial datasets with segmentation masks do not exist in the file, so we build one. We will annotate about 3,000 frames from the real sequences of the pixel-aligned RGB-NIR dataset with instance masks for people, vehicles, and furniture. Frames will be drawn to over-represent its dark, glare, and shadow conditions. A second set of about 2,000 nighttime KAIST frames will receive person masks, seeded from the existing boxes. Annotators label on a fused visualization so that labels are not biased toward either modality. Following the uncertainty-band idea in Polyper, each boundary pixel is additionally tagged with RGB local contrast (gradient magnitude across the ground-truth edge) and auxiliary-channel local contrast. The small hybrid-endoscope set (12 labeled frames) serves only as a qualitative out-of-domain check.

Parallax simulation. The RGB-NIR dataset includes LiDAR depth, so we can reintroduce parallax exactly as the standard two-view relation predicts. For baseline vector $\mathbf{t}$ and per-pixel depth $Z$, an auxiliary pixel moves from $\mathbf{x}_1$ to

$$\mathbf{x}_2 = \mathbf{H}_\infty \mathbf{x}_1 + \frac{\mathbf{K}\mathbf{t}}{Z},$$

where $\mathbf{H}_\infty$ is the homography at infinite depth and $\mathbf{K}$ is the camera intrinsic matrix. We render the auxiliary image from a virtual camera at baselines from 0 to about 10 cm, using z-buffering so that occlusion holes appear as they would in a real side-by-side rig. We then re-register it to RGB in three ways, all operating on the warped auxiliary image and none using the ground-truth depth. The first is a global homography. The second is monocular-depth-based warping, which does not assume depth is known. The third is dense cross-modal optical flow. Together these give the realistic software pipelines a skeptic would propose. KAIST has no depth, so it receives only depth-independent shifts, which serve as a lower bound on the harm.

Model. The backbone is a standard ResNet-encoder U-Net with two modality streams. The single new ingredient is contrast-routed skips. At each skip scale, a small gate takes the two streams' shallow features plus a fixed local-gradient-magnitude map for each modality. It outputs a per-pixel softmax over modalities, and that softmax weights which stream's features pass to the decoder. Resolution-mismatched inputs such as time-of-flight are upsampled before the gate, and the gate learns to down-weight their blur. Training uses Niu et al.'s combined segmentation-plus-contour loss. A Polyper-style boundary refinement stage can be attached on top. We vary its presence as an experimental factor rather than treating it as part of the method.

Baselines

The naive reference point is an RGB-only U-Net with the same encoder and loss. Its low-RGB-contrast boundary stratum defines the problem size. If the stratum is small, the whole question is minor, and we will report that.

The obvious fusion baseline is the same U-Net with the auxiliary channel concatenated at the input. Its contour loss and backbone are identical to ours, and it gets the same tuning budget. It is the direct competitor to contrast-routed skips and the most likely to tie with them.

The strongest existing multi-modal method is CRM with self-distillation, whose reported number is 61.2% mIoU on MFNet (Complementary Random Masking). We will reproduce it on MFNet to validate our implementation within a point of that figure. We will then train it on our coaxial splits with the same number of epochs, the same hyperparameter search budget, and three seeds. FEANet is included as a lighter attention-based fusion baseline. There is no reported result on Lacuna for any segmentation method on the pixel-aligned RGB-NIR dataset or on mask-annotated KAIST, so all numbers on our splits are new.

The strongest RGB-only boundary method is Polyper, reported at 86.51% mDice on ETIS (Polyper). It is trained on RGB alone, to measure how far algorithmic refinement alone can go. MISNet serves as a second single-modality refinement baseline.

The ablations isolate the key ingredient. The first removes the gradient-magnitude inputs to the gates, leaving them learned only. The second replaces gating with a fixed 50/50 average. The third adds or removes Polyper refinement on top of each fusion variant.

All methods share data, input resolution, augmentation, and a GPU-hour cap. For the proposal to count as "beating" a baseline, it must show a boundary F-score improvement at 2-pixel tolerance in the low-RGB-contrast stratum that exceeds the seed-to-seed spread. Overall mIoU must also stay within noise of the best competitor. An aggregate-mIoU win alone does not count.

Why not just…?

Why not just concatenate the channels into an off-the-shelf U-Net with a boundary loss? This is the real threat. On perfectly aligned data, concatenation may well match gated skips, because the network can learn modality weighting in its first layers. The CRM paper's "modality drop" argument, that networks over-rely on the dominant sensor, was made for RGB-thermal data. It is suggestive for our setting but not proof. Our stratified comparison settles it. If concatenation ties in the low-RGB-contrast stratum at zero baseline, we drop the architectural claim. The paper then becomes the tolerance study, which we would still regard as the more important result. Where we do expect gating to separate from concatenation is under partial misregistration and with the 64×48 time-of-flight channel, where a pixel-level choice to ignore a blurred or shifted sensor should matter.

Why not just use side-by-side cameras and register in software? Kim and Baek describe the circular dependency: aligning the images needs depth, and estimating depth needs aligned images. We do not rely on that argument alone. The parallax simulation, with three realistic re-registration pipelines, measures exactly how much boundary accuracy each baseline length costs. If flow-based re-registration keeps the low-contrast gain up to a practical baseline, then coaxial hardware is unnecessary for this task, and we will say so. Our comparison has one caveat that favors coaxial hardware. Both simulated arms inherit the beam splitter's halving of light per sensor, so a real side-by-side rig would have a less noisy RGB image. We will quantify this by adding matched sensor noise to the zero-baseline arm and report both arms.

Why not just use better single-modality boundary refinement? Polyper, reverse attention, and shape priors sharpen edges when a faint contrast signal is present. They cannot create a signal the sensor never recorded. The stratified metric tests this directly: we predict Polyper on RGB alone gains little in the stratum where RGB contrast is near zero. If it gains as much as fusion does, the physical-contrast premise is wrong.

Why not just prompt a large foundation segmenter? CaPro reports that SAM needs structure-specific prompting to handle thin, low-contrast boundaries. Foundation segmenters also take RGB input, so they face the same missing-signal limit in the zero-contrast stratum. We include SAM prompted with ground-truth boxes, on RGB and on an RGB-NIR false-color image, as an upper-bound-style reference rather than a trained competitor.

Proposed Experiments

The first decisive experiment is the stratified gain analysis at zero baseline. It reports boundary F-score at 1, 2, and 3 pixels, 95% Hausdorff distance, and mIoU, for every method, in RGB-contrast quartiles. It confirms or kills the claim that the auxiliary gain concentrates at low-contrast boundaries.

The second is the tolerance curve. We plot the low-contrast-stratum boundary F-score against simulated baseline, from 0 to 10 cm, for concatenation, gated skips, and CRM, under each re-registration pipeline. We report the misregistration, in pixels, at which the fused model falls to the RGB-only level. We expect that crossover near the metric tolerance, with pixels adjacent to occlusions crossing first.

The third is a two-by-two factorial crossing fusion (none, gated) with refinement (none, Polyper), evaluated separately in the stratum where the auxiliary channel has contrast and the stratum where neither sensor does. It tests whether algorithmic refinement becomes redundant once physical contrast exists.

A final transfer check applies the KAIST-trained person model to the hybrid-endoscope frames, qualitatively only. This probes whether contrast routing generalizes to a range-plus-color sensor pair with a 10× resolution gap. There we will compare against the reported AND-consolidation F-score of 0.73 from Haase et al. without claiming a controlled win on 12 frames.