A robot or phone often carries two cameras of very different quality. One is a sharp color camera. The other is a coarse, noisy depth sensor, such as a Time-of-Flight unit that measures distance by timing reflected light. Recovering a sharp depth map from the coarse one, guided by the color image, is called guided depth super-resolution. It is a well-studied problem, and it is usually treated as if both images were already on the same computer. In practice the heavy reconstruction often runs on a server, so both images must first cross a noisy multi-antenna wireless link. This note argues that the color image should not be sent whole. Color helps depth only near depth edges. Everywhere else it hurts, because printed patterns and wallpaper stripes get copied into the depth map as fake relief.
We propose to transmit the coarse depth map plus only a thin "boundary layer" of color pixels. This layer is a narrow band around the depth discontinuities, and the receiver can locate it from the depth map it already decoded. We send these two streams over a wireless channel that has been split into parallel sub-channels of identical quality. The core claim has two parts. First, at the same number of wireless channel uses, boundary-layer guidance gives lower depth error than sending full color. Second, it removes the main cause of texture copying at the source rather than filtering it out afterwards. The equal-quality channel split reduces the sender's resource decision to a single number: how many channel uses go to depth and how many go to boundary color.
Where this sits
The three ingredients come from separate literatures, and each plays a different role here.
The reconstruction problem and its central failure come from guided depth super-resolution. The survey of guided depth super-resolution names two persistent artifacts. Texture copying moves color patterns with no depth relief into the depth map. Edge bleeding smears depth across blurred or misaligned edges. Recent networks fight these artifacts inside the network's features. SUFT builds an uncertainty map by flipping features and lets color in only where uncertainty is high. SGNet injects the high frequencies missing from the coarse depth spectrum. SSDNet separates shared structure from modality-specific content on a hypersphere. IGAF refines color guidance with attention over several rounds and reports overfitting beyond three rounds. These papers motivate the proposal. Each one needs extra machinery to stop a network from trusting interior color, which suggests that learned fusion does not ignore texture on its own. All of them assume the full color image is available for free. We remove that assumption and ask what the guidance is worth once it costs bandwidth.
The boundary-layer idea comes from layered image decomposition. View Synthesis for Advanced 3D Video Systems split scenes into a main layer of smooth depth and dedicated boundary layers along depth edges, which removed warping artifacts. HiDAnet cut depth maps into foreground, midground and background intervals using multi-level Otsu thresholding, an automatic method for picking intensity cut points. We reuse both ideas, but for a different purpose. In our system the layering does not serve editing or saliency detection. It decides which color pixels are worth transmitting at all. We deliberately avoid amodal completion, meaning the filling-in of hidden background as in MULAN or Move Anything with Layered Scene Diffusion. Filling in hidden background invents content, and invented depth is worse than missing depth.
The channel model comes from multi-antenna (MIMO) channel decomposition. ViT-MIMO is the closest existing system. It splits a MIMO channel into parallel sub-channels with singular value decomposition (SVD), which gives sub-channels of unequal strength. It feeds the per-sub-channel noise levels to a vision transformer, which learns to route important image patches onto the strong sub-channels. It transmits single-modality RGB, has no notion of layers or of a downstream depth task, and assumes perfect channel knowledge. GMD-Based Hybrid Beamforming supplies the alternative split we use: the geometric mean decomposition (GMD). It factors the channel matrix $H$ as
$$H = Q R P^{*},$$
where $Q$ and $P$ are unitary rotations applied at the receiver and transmitter, and $R$ is upper triangular. Every diagonal entry of $R$ equals $\bar\sigma = (\prod_{i=1}^{K}\sigma_i)^{1/K}$, the geometric mean of the $K$ nonzero singular values $\sigma_i$ of $H$. After the receiver applies $Q^{*}$ it observes $z = R s + \tilde n$, where $s$ is the vector of transmitted symbols and $\tilde n$ is rotated noise. Every stream therefore has the same strength. The price is known, triangular leakage from later streams into earlier ones.
Core Claim
The claim is specific. Fix a budget of $k$ complex channel uses. Spend $k_D$ of them on the coarse depth map and $k_B = k - k_D$ on color restricted to a band $M$ around depth edges. Then a receiver-side super-resolution network reaches lower high-resolution depth error than the same network given full color and coarse depth under the same budget. The gap should widen as the budget shrinks.
The band $M$ is computed from the decoded coarse depth map. Its width is roughly the upsampling factor, because a coarse depth edge pins the true edge only to within one coarse pixel. Inside that band, color edges are what locate the true boundary. Both sender and receiver compute $M$ from depth, so the mask costs almost nothing to transmit. The one complication is that the receiver sees a noisy depth map, so its mask may differ slightly from the sender's, and we measure that mismatch directly.
A second consequence is that the bandwidth adapts to content. A scene of large flat walls has a short boundary and needs few color symbols. A cluttered desk needs many. A fixed-ratio learned encoder like ViT-MIMO cannot vary its spending this way.
The equal-gain channel is what keeps the allocation simple. Under GMD the sender's whole decision is the scalar split $k_D/k$, conditioned on one number, $\bar\sigma^2/\sigma_n^2$, where $\sigma_n^2$ is the noise variance. Under SVD the split interacts with which content lands on which unequal sub-channel. We do not claim GMD is free. With analog, non-digital symbols the receiver cannot cleanly subtract decoded streams, so the triangular leakage adds noise. Measuring that cost is part of the program.
Research Program
The first stage is noiseless and isolates the source-side idea. We retrain SUFT, SGNet and IGAF with their released settings. Each receives color multiplied by the band mask $M$ instead of full color. We sweep the band width and build $M$ in two ways: from Canny edges on the upsampled coarse depth, and from boundaries between Multi-Otsu depth layers as in HiDAnet. The question is whether masked guidance matches full guidance and whether it lowers error in textured flat regions. To test the second point we also report error restricted to pixels that lie far from depth edges but on strong color edges, which are the places where texture copying happens.
The second stage adds the channel. A transformer encoder in the style of ViT-MIMO maps the coarse depth map and the band-masked color to $k_D$ and $k_B$ symbols. The symbols travel over simulated $2\times2$ and $4\times4$ Rayleigh MIMO channels, the standard random-fading model, at 0–22 dB, the range ViT-MIMO used. We precode with either SVD or GMD. The GMD decoder receives $z$ and $R$ and learns to undo the triangular leakage. We compare it with plain linear back-substitution so that the cost of analog leakage is visible. A super-resolution network at the receiver is trained end to end on depth loss.
The third stage stresses the channel model. We add channel-estimation error at the transmitter and coarse phase shifters of the kind discussed in the GMD hybrid-beamforming paper, and ask whether a model conditioned on a single scalar SNR degrades more or less gracefully than one conditioned on a full noise heatmap.
Baselines
The naive reference sends no color. All $k$ channel uses go to the coarse depth map, and the receiver upsamples it with bicubic interpolation and, separately, with the classical guided filter. This shows what color guidance is worth at all. The survey reports guided-filter RMSE of 5.41 against 1.56 for the learned AHMF at ×8 on Middlebury, which sets the scale of the gap in the noiseless case.
The strongest existing pipeline pairs full-color transmission with the best reconstructor. We extend ViT-MIMO to four input channels (color plus coarse depth) with SVD precoding and its noise-heatmap conditioning. We decode the result and pass it to SUFT, SGNet and IGAF, retrained on decoded inputs. This is the strongest competitor because it gives the learned encoder full freedom to discover on its own that interior color is useless.
Its noiseless ceilings are the reported numbers. On NYU v2 these are IGAF's RMSE of 1.12, 2.48 and 5.00 at ×4, ×8 and ×16, and SUFT's 1.12 cm at ×4. On RGB-D-D they are SSDNet's 2.92 at ×16 and SUFT's 5.41 cm. Units and protocols differ across papers, so we compare each method only to its own setting. No reported result was found on Lacuna for any RGB-D transmission over MIMO, so the wireless numbers will be new. Every system gets the same channel-use budget, the same training data, the same receiver network and the same hyperparameter-search budget.
A third baseline follows the separation principle: separately designed compression and channel coding. Color and depth are compressed with the BPG image codec, sent at the channel's theoretical capacity, decoded, and super-resolved. This is the idealized digital baseline ViT-MIMO beat by up to 3.5 dB PSNR on CIFAR10 at bandwidth ratio 1/6.
A fourth baseline runs super-resolution on the sender and transmits the finished high-resolution depth map. It serves as an upper reference for when the device has enough compute, and we report its on-device cost alongside its accuracy.
Three ablations isolate the key ingredient. In the first, a random mask with the same area replaces the boundary mask, which tests whether the gain comes from spending fewer color symbols rather than from choosing the right pixels. In the second, a mask built from color edges alone replaces the depth-derived mask, which tests whether depth is needed to decide where color matters. In the third, SVD replaces GMD with everything else fixed, which isolates the equal-gain channel.
Beating the strongest baseline means one of two outcomes at matched channel uses, holding across at least two SNR levels and two scales on NYU v2 and the real-sensor RGB-D-D set. Either the full-image depth RMSE is lower, or the RMSE in texture-copying regions is lower with the full-image RMSE no worse.
Why not just…?
Why not just let an end-to-end encoder learn what color to send? This is the real threat. A joint encoder trained on depth loss could in principle learn to discard interior color. The evidence against it is indirect but consistent. SUFT, SGNet, SSDNet and IGAF each needed explicit machinery to stop networks from trusting interior color, and IGAF saturated when it tried to solve the problem with more depth of processing. A fixed-ratio encoder also cannot vary its symbol count with scene content. The four-channel ViT-MIMO baseline settles this directly. If it matches us at every budget, our contribution shrinks to a diagnostic finding plus content-adaptive rate, and we will report it that way.
Why not just run super-resolution on the device and send the sharp depth map? When on-device compute is cheap this may win, and the fourth baseline is included to find out. Our case rests on two points. The strongest reconstructors are heavy. Sending a sharp depth map spends symbols on smooth interiors, which the coarse map already describes well. We will report the accuracy-versus-device-compute curve rather than assume an answer.
Why not just use SVD with learned power allocation, as ViT-MIMO does? SVD is the capacity-optimal split. ViT-MIMO showed that removing SVD precoding cost up to 2.9 dB, but it also had to spend transformer capacity learning which patches go to which sub-channel. GMD may lose some of that efficiency through analog leakage between streams. The SVD/GMD ablation, run with both learned and linear back-substitution decoding, measures exactly this trade, both with perfect channel knowledge and with estimation error.
Why not just compress digitally and use standard channel codes? ViT-MIMO's 3.5 dB gain over the idealized BPG-plus-capacity baseline, and the sudden collapse of digital schemes when the channel drops below the coded rate, both argue against it. We still include it, because depth maps compress unusually well and could narrow that gap.
Decisive Checks
The noiseless masked-guidance experiment comes first and can end the project cheaply. If band-masked color loses clearly to full color even with no channel noise, the source-side claim is false. The likeliest cause is thin structures, such as chair legs, that disappear from coarse depth at ×16 and so never enter the band. In that case we would test a low-rate color thumbnail added to the band before going further. If masked guidance holds up, the matched-budget comparison against four-channel ViT-MIMO with SUFT, IGAF or SGNet decides the main claim. The SVD/GMD ablation under channel-estimation error then decides whether the equal-gain channel is worth its leakage cost.
References
- Guided Depth Map Super-Resolution: A Survey
- Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution
- SGNet: Structure Guided Network via Gradient-Frequency Awareness for Depth Map Super-resolution
- Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution
- IGAF: Incremental Guided Attention Fusion for Depth Super-Resolution
- View Synthesis for Advanced 3D Video Systems
- HiDAnet: RGB-D Salient Object Detection via Hierarchical Depth Awareness
- MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation
- Move Anything with Layered Scene Diffusion
- Vision Transformer for Adaptive Image Transmission over MIMO Channels
- GMD-Based Hybrid Beamforming for Large Reconfigurable Intelligent Surface Assisted Millimeter-Wave Massive MIMO