Width-Aware Pixel Bypasses for Thin Structures in Sparse-View Radiance Fields

Wires, fence slats, plant stems and ship rigging are where 3D reconstruction from photographs visibly breaks. A neural radiance field is a network that maps a 3D point and viewing direction to a color and a density, and renders images by summing those values along each camera ray. When such a model meets a structure thinner than a pixel, it tends to dissolve it into a gray fog, cut it into disconnected pieces, or glue it to the background. The detail-preservation literature has a standard remedy for losing fine structure: skip connections. These are pathways that carry high-resolution input directly to the output instead of forcing it through a compressed bottleneck. Generalizable radiance fields, which predict a scene from a few input photos without per-scene training, already use this remedy. IBRNet blends raw source-image pixels instead of regressing color from network weights. Thin structures still fail.

We argue that the bypass is being handed the right pixels and then misused at the compositing step. A pixel that sees a sub-pixel wire records a mixture: the wire's color weighted by the fraction of the pixel it covers, plus the background. That fraction depends on how far away the camera is and on its resolution. Standard volume rendering can only express it as a view-independent semi-transparent density, which is fog. We propose to give each rendered sample an explicit physical width. The model converts that width into a per-view pixel coverage using the camera's focal length and the sample's depth. It infers the width feed-forward from whichever source views see the structure at multiple pixels wide, and it pulls the structure's color through a full-resolution pixel bypass. In short, a close-up source photo tells the network how wide the wire is and what color it is, and simple projective geometry then says how much of each far-away target pixel the wire should occupy.

Core Claim

The claim is that thin-structure failure in sparse-view radiance fields is mainly a compositing-model failure, not a feature-resolution failure. Fixing it requires two things together. The first is a bypass that delivers native-resolution source colors to each ray sample. The second is a coverage term in which opacity scales with projected width, so that one latent width per point is consistent across all views. Concretely, for a ray sample $i$ at camera depth $z_i$, with predicted world-space width $w_i$, predicted solid-ness $o_i \in [0,1]$, and target camera focal length $f$ in pixels, we replace the usual sample opacity with

$$\alpha_i = o_i \cdot \min\!\left(1, \frac{f\, w_i}{z_i}\right),$$

and composite as usual: $\hat C = \sum_i T_i \alpha_i c_i$, with transmittance $T_i = \prod_{j<i}(1-\alpha_j)$. The color $c_i$ is a learned blend of raw source pixels fetched by projecting the sample into each source view.

The ratio $f w_i / z_i$ is simply the structure's projected width in pixels. Bulk surfaces predict large $w_i$, so the clamp saturates and the formula reduces to ordinary opaque compositing. A wire seen from far away predicts small $w_i$ and contributes partial coverage. That partial coverage shrinks correctly as the camera recedes, instead of staying fixed as a density-based fog would. The testable prediction follows from this. Baselines should show thin-structure error that rises sharply once projected width drops below about one pixel. Our model should degrade gracefully along that axis.

This draws on two papers in different roles. Thin Structures in Image Based Rendering motivates the design. It shows that multi-view stereo fails on sub-pixel railings and wires and binds them to the background. It recovers them by treating alpha as the probability that a thin structure lies along the ray, combined with multi-layer graph-cut segmentation driven by cross-view color variance. That pipeline needs user-clicked proxy planes and cylinders, works with discrete depth layers, and fails when foreground and background colors match. We keep its core insight, that partial alpha means partial coverage, but make coverage a continuous, geometry-derived quantity predicted without user input.

IBRNet supplies the method. It contributes the shared 2D U-Net, the per-ray transformer that reasons about ordering along the ray, and the blending of raw source colors. That blending is exactly the spatial-detail skip connection the second research direction recommends. What IBRNet does not do is decouple a sample's opacity from the target view's footprint. That decoupling is our addition.

Research Program

The model is built on an IBRNet-style backbone. A shared U-Net extracts features from a working set of nearby source views. Each ray sample is projected into those views, where the model reads both the U-Net features and the raw source pixel colors at full native resolution. A ray transformer then reasons along the ray.

The heads are changed in three ways. First, the density head is replaced by two outputs, solid-ness $o_i$ and log-width $\log w_i$. Second, the aggregation step receives, for each source view, the projected width $f_k w_i / z_{ik}$ that the current hypothesis implies in that view, where $f_k$ is view $k$'s focal length and $z_{ik}$ the sample's depth in view $k$. This lets the network check whether a high-variance thin feature in a close view is consistent with a faint smear in a far view. Third, the color blend gives more weight to source views where the projected width exceeds one pixel. Those views observed the structure's true color unmixed, which is the only place an unmixed color exists.

Training uses the same photometric loss as IBRNet, applied to target views at deliberately varied distances and resolutions. The coverage term only receives a useful gradient when the same structure is seen at different projected widths. To provide that, we add a procedurally generated synthetic corpus of wires, meshes, lattices and branching stems at controlled widths in front of textured backgrounds, rendered from camera rings at several radii. This mirrors the role synthetic root data plays in Adversarial Large-Scale Root Gap Inpainting. That paper used a skip-connected U-Net to reconnect occluded root segments and measured success as a drop in disconnected segments per patch. We borrow that connectivity metric. We do not borrow its 2D adversarial machinery, because our missing structure is not occluded but sub-pixel.

The corpus is mixed with the multi-scene training data IBRNet already uses: Google Scanned Objects, RealEstate10K and Spaces. The analysis half of the program is a "detail ledger." On synthetic scenes with known wire masks and widths, we measure how much thin-structure signal survives each stage of the pipeline: source pixels, U-Net features, epipolar samples, aggregation, and the final compositing. This tests directly whether compositing, rather than encoding, is where the loss occurs. If the ledger shows the loss happening in the encoder instead, the central claim is wrong and the proposal reduces to a stronger-encoder study. We will report it as such.

Baselines

The naive reference point is per-scene NeRF (NeRF), trained on the same sparse views we give our model. On the full dense Realistic Synthetic set it reports 31.01 dB PSNR. We will quote that figure but compare only under matched view counts, because dense-view numbers are not comparable to a sparse setting.

The strongest existing competitor is IBRNet. It already contains the high-resolution bypass and ray-level reasoning, so it isolates exactly what our coverage term adds. On Real Forward-Facing (LLFF) it reports 25.13 dB PSNR zero-shot and 26.73 dB after per-scene fine-tuning, against 26.50 dB for NeRF. We will retrain IBRNet from its public recipe on our expanded training mixture, including the synthetic thin-structure corpus, so that the data advantage is removed. It gets the same working-set size, sample count, GPU budget and number of fine-tuning steps. Beating it means two things: a clear improvement on thin-structure-masked metrics and connectivity in the sub-pixel regime, and no loss in whole-image PSNR on LLFF and Blender. A whole-image gain alone would not count.

Two other generalizable methods supply different detail-preservation mechanisms. pixelNeRF (pixelNeRF) uses a downsampling ResNet encoder with averaged multi-view features. It reports 23.72 dB on single-view ShapeNet chairs, a different task, and is included to show what happens without any bypass. MVSNeRF (MVSNeRF) uses a 3D U-Net over a plane-swept cost volume, a structure that compares source features across a stack of candidate depths. It is included to test whether voxel-resolution skip connections suffice. On the file's records, its strength is speed: 15-minute fine-tuning versus 10.2 hours for NeRF. No PSNR figure is recorded for it on these benchmarks.

As a per-scene upper reference we include NeXT (NeXT). It re-injects positional encodings into every transformer block along a ray and reports 34.40 dB PSNR on Blender. It is not a sparse-view method, so we treat it as a ceiling rather than a target.

Three ablations isolate the key ingredient. In the first, coverage is replaced by a free per-view opacity correction. This tests whether physical width matters or merely extra capacity. In the second, the width input to aggregation is removed. In the third, raw-pixel blending is replaced by regressed color while the coverage term is kept. This tests whether width alone, without the bypass, can recover thin structures.

Why not just…?

Why not just restore the rendered views in 2D afterwards? Off-the-shelf restoration such as NeRFLiX (NeRFLiX_CVPR2023), or 2D super-resolution, sharpens what the renderer produced. A wire already rendered as uniform fog carries no recoverable structure, so the restorer must hallucinate one per frame. Super-NeRF documents this problem: super-resolution is underdetermined, and view-independent sharpening is inconsistent across views. Its fix is costly per-scene mutual training of the super-resolution network and the radiance field. This alternative is a real threat on whole-image perceptual scores, so we will run NeRFLiX on top of IBRNet as an additional baseline. The deciding evidence will be the connectivity count and cross-view flicker on thin-structure masks, where post-hoc restoration has no 3D constraint.

Why not just use a higher-resolution encoder, ReferenceNet-style? Animate Anyone preserves textures by injecting multi-scale features from a parallel U-Net. IBRNet already goes further and routes raw pixels themselves, so encoder resolution is not the bottleneck it is in pixelNeRF. Our detail ledger and the regressed-color ablation test this directly. If thin structures survive into the aggregated features but vanish after compositing, more encoder capacity cannot help.

Why not just use per-scene NeRF with more samples and higher positional-encoding frequencies? With dense views this may genuinely work. NeXT's large Blender gain shows per-scene capacity can recover a lot of detail. Our regime is a few views from varied distances, though, and there the opacity a density-based model assigns to a sub-pixel wire is fixed by the density field itself, regardless of how far away the camera is. Adding samples does not change that. We will settle the question with a view-count sweep, reporting the count at which per-scene NeRF and NeXT catch up.

Why not just use cone-based anti-aliased rendering? Integrating over a pixel's cone footprint, as Mip-NeRF does, also makes appearance depend on distance. This is the closest conceptual competitor. The file gives only its render time (2.45 s per image) and no reported result on Lacuna for its thin-structure accuracy. It is per-scene and learns footprint effects implicitly, whereas we predict an explicit width feed-forward and use it to choose which source pixel is unmixed. We will include it on the per-scene track. If it matches us on the projected-width sweep under sparse views, our contribution narrows to generalization, and we will say so.

Proposed Experiments

The decisive experiment is the projected-width sweep on the synthetic thin-structure corpus. Thin structures of known width are rendered at target distances spanning projected widths from about 4 pixels down to 0.2 pixels. We plot masked PSNR and LPIPS (a learned perceptual distance), disconnected-segment count, and background-bleed error against projected width for every method. The hypothesis predicts a crossover near 1 pixel, below which IBRNet and MVSNeRF collapse while our model does not. No crossover would falsify the claim.

Transfer to real data is tested on three sets. On LLFF we focus on the orchids and horns scenes, using hand-drawn thin-structure masks. On the Blender scenes we focus on ship rigging and the microphone mesh. We will also use the nine scenes of Thin Structures in Image Based Rendering if the authors release them. That paper reports no PSNR, so there we can compare only qualitatively and on connectivity.

Throughout, whole-image PSNR, SSIM and LPIPS on LLFF and Blender must not regress relative to retrained IBRNet. This guards against a coverage term that helps wires but leaks transparency into solid surfaces. Finally, a temporal-consistency check renders orbits and receding dolly shots and measures frame-to-frame flicker inside thin-structure masks. That is the setting in which a fixed-fog representation and a width-aware one should look most different.