Robots and vehicles keep running into glass. A camera can recognize a glass door by its frame, reflections and surrounding context, but only when the scene is well lit. A laser range sensor (LiDAR) is unaffected by lighting, but its beam mostly passes through glass and reports the wall behind it. Current fusion detectors treat the laser's first echo as the ground truth for where objects are. Glass breaks that assumption, so those detectors either see open space or inherit the camera's failures in the dark.
This note proposes the opposite division of labor. The laser stays the lighting-independent anchor, but it is read differently: the model looks at each beam's full set of echoes rather than its single best point. A beam that passes through glass often produces a faint early echo before the strong return from the background, and its intensity spikes when it hits a pane head-on. The camera then refines that anchor, and its influence is scaled by an explicit, measured estimate of how trustworthy the image is at that lighting level and distance. The claim to test is that this combination can detect and localize glass in conditions where no current method works: dark scenes, glare, and panes at tens of meters where few beams land. It should also never do worse than the laser alone when the camera is useless.
Core Claim
The physical signals of transparency are already present in multi-return LiDAR. 3DRef shows that obstacles seen through glass produce a higher share of second and third returns than opaque walls. TOPGN exploits the sharp intensity peak glass gives at near-perpendicular incidence. These cues are weak and partial, though. In 3DRef, adding the return index as an extra input channel gained only about 0.5% mIoU, and TOPGN's cue vanishes for tilted or curved panes. Camera cues are strong but fragile. Retraining EBLNet on 3DRef lifts glass mIoU to 86.71%, yet lighting is the known failure mode of every image-driven transparent-object method. FDCT names extreme lighting as its open problem, and ClearGrasp reports caustics being mistaken for boundaries.
The weak LiDAR gain in 3DRef comes from flattening a beam's echo sequence into a per-point channel. Our hypothesis is that two changes recover it: model the whole echo profile of each beam as a unit, and let camera features attend to these beam units with a reliability-controlled weight. The resulting detector should degrade gracefully from camera-quality accuracy in good light to LiDAR-only accuracy in darkness, and at long range where beams thin out.
This differs from the closest work in three ways. TransFusion seeds object queries from first-return LiDAR features and ignores echo structure. 3DRef evaluates LiDAR and RGB separately and lists fusion as future work. GraspALL adapts modality weighting to light, but it assumes the depth channel is reliable, which is false for glass.
Research Program
Beam tokens. We would replace per-point inputs with one token per laser beam. Each token packs the beam's ordered returns, giving range, intensity and return index for each echo. It also carries an incidence-angle estimate from a locally fitted plane, and a flag for beams with no return at all.
Echo-seeded queries. A lightweight sparse encoder (a small point transformer over beam tokens) produces glass and mirror proposals. These proposals are seeded wherever echo structure deviates from the opaque-surface pattern, not only where a strong first return exists. This reverses the "LiDAR-first" failure that Benchmarking the Robustness of LiDAR-Camera Fusion documents for TransFusion, PointAugmenting and MVX-Net. In those models, missing returns collapse accuracy toward zero.
Soft camera cross-attention. Proposals cross-attend to image features using TransFusion's spatially modulated attention. This is a Gaussian-weighted soft association that the TransFusion paper shows is robust to calibration error. Hard projection would be wrong here: a refracted point measures the background, so projecting it pastes background features onto the pane. The image backbone supplies frame edges, reflections and semantic context. GlassSemNet shows such context localizes glass when edge cues are missing. That context is also what confuses mirrors with windows, and echo profiles should resolve that confusion.
Reliability gate. The camera term is multiplied by a scalar gate predicted from cheap measured quantities: image noise level and saturated-pixel fraction in the projected region, plus range to the proposal. GraspALL indexes lighting with parametric luminance curves. We borrow the idea of a quantitative lighting reference but drive the gate with measured image statistics, so the system needs no lux meter.
Training regime. We train with controlled photometric corruption so the gate sees the full range of conditions. Corruption covers exposure reduction with a shot-plus-read noise model, gamma and glare overlays, while the LiDAR input stays untouched, which is physically correct. We also apply beam subsampling to emulate the angular sparsity of distant panes.
Outputs. The model produces two things. One is per-point labels (glass, mirror, through-glass background, other). The other is a corrected first-surface range for beams that passed through glass, meaning the pane's actual location. That second output is what a planner needs.
Baselines
Do-nothing reference. A standard four-channel LiDAR segmentation network trained on 3DRef, with no return index and no camera. This sets the floor that any fused system must never drop below. Absolute LiDAR-only mIoU values for 3DRef are not given in the file; we only have the roughly 0.5% gain from the return channel. We would rerun this baseline ourselves on identical splits.
LiDAR-only with return channel. The 3DRef five-channel variant, which is the strongest existing lighting-immune method on this data. We would also run a TOPGN-style intensity-layer detector reimplemented on 3DRef's LiDARs. TOPGN reports a 12.74% F-score improvement and 38.46% lower MAE, but only on its own Turtlebot scenarios, not on 3DRef. Beating these baselines in good light is easy for any fusion method. The real test is matching them in darkness.
Strongest camera competitor. EBLNet retrained on 3DRef, reported at 86.71% glass mIoU in the 3DRef paper. Its 2D masks are lifted to points by projection. We would rerun it on our splits so the comparison uses identical data, since the reported number is a 2D image metric.
Strongest fusion competitor. A TransFusion-style detector adapted to segmentation and given the same five-channel points. It keeps first-return query seeding and has no gate. It runs with the same image backbone, training schedule, corruption augmentation and hyperparameter search budget as our model, so the remaining difference is how LiDAR is read and how camera weight is controlled.
Ablations. Three ablations isolate the key ingredients: beam tokens replaced by per-point tokens with return-index channels, a gate frozen to a constant, and echo-seeded queries replaced by first-return seeding.
What counts as winning. Beating the baselines means meeting all of the following:
- Exceed the fusion baseline by a clear margin in the dark and long-range strata.
- Stay within noise of EBLNet in normal light.
- Never fall below the five-channel LiDAR baseline under any corruption level.
- Reduce glass–mirror confusion relative to EBLNet.
Why not just…?
Why not just drop the camera and use LiDAR echoes, as TOPGN does? This is the honest null hypothesis for dark scenes. Our design concedes it: in darkness we only promise to match it. The evidence that it falls short elsewhere comes from both sources. TOPGN's own analysis says the intensity cue disappears on tilted or curved glass, and 3DRef reports Ouster sensors give few distinct returns on glass. The range-and-incidence stratified evaluation below would show where LiDAR-only fails and whether the camera recovers those cases.
Why not just fuse the outputs of a retrained image segmenter and a LiDAR segmenter at the end? This is a real threat. Averaging EBLNet masks with five-channel LiDAR predictions, with a confidence-based weight, may capture much of the benefit. We include it as a baseline with the same gate inputs. If it matches our model, the contribution shrinks to the beam tokenization and the evaluation protocol, and we would report that. We expect it to lose specifically on mirrors and through-glass relabeling, where the camera needs to know which echo it is looking at.
Why not just put a low-light enhancer in front of an existing fusion model? Diffusion restorers like EndoUIC are slow for closed-loop use. Enhancement also cannot recover edges that have dropped below the noise floor, and GraspALL argues that decoupled enhancement does not fix downstream perception. We would still run a fast enhancer plus the fusion baseline at matched latency to measure this directly.
Why not just feed the return channels into TransFusion? That is exactly the fusion baseline. If it closes the gap, beam-level modeling is unnecessary.
Proposed Experiments
All training and testing use 3DRef's aligned multi-return LiDAR and panoramic RGB, split by sequence. Results are stratified along three axes:
- Lighting: five photometric corruption levels, from clean to near-black, plus glare overlays.
- Range: distance bins within the indoor sequences.
- Sparsity: beam subsampling at factors emulating range beyond what the indoor sequences contain.
The metrics are point-level glass and mirror IoU, the glass↔mirror confusion rate, and first-surface range error on through-glass beams. The range error is computed where 3DRef's annotations permit recovering pane geometry; otherwise it is reported on a hand-labeled subset. Synthetic darkening only perturbs the camera, so it is a clean test of the gate. We would add a small real capture with room lights off, on a Livox triple-return sensor, to check that the synthetic noise model does not flatter the method.
Graceful-degradation curve. The first decisive check plots accuracy against corruption level. The fused model must interpolate between the EBLNet and LiDAR-only curves and never dip below the LiDAR floor. Its failure mode would be the mirror image of the LiDAR dependency trap, and it should not have it.
Ablation in the hard strata. The second check is the beam-token ablation in the far-range and oblique-incidence strata. If beam tokens do not beat per-point return channels there, the central hypothesis is wrong.
Late-fusion comparison. The third check is the comparison against late fusion. Losing it would redirect the work toward the cheaper design.
References
- 3DRef: 3D Dataset and Benchmark for Reflection Detection in RGB and Lidar Data
- TOPGN: Real-time Transparent Obstacle Detection using Lidar Point Cloud Intensity for Autonomous Robot Navigation
- FDCT: Fast Depth Completion for Transparent Objects
- Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation
- TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
- GraspALL: Adaptive Structural Compensation from Illumination Variation for Robotic Garment Grasping in Any Low-Light Conditions
- Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection
- Exploiting Semantic Relations for Glass Surface Detection
- EndoUIC: Promptable Diffusion Transformer for Unified Illumination Correction in Capsule Endoscopy