Reliability-Driven Feature Correction for Hyperspectral and LiDAR Classification

Lianming Zhang, Xiaolu Wang, Qiya Song, Jinyang Liu, Renwei Dian, Xudong KangarXiv 2026 · arXiv

TL;DR

ReDF is a new framework for fusing Hyperspectral (HSI) and LiDAR data that uses class-prototype evidence to quantify the "reliability" of each modality at every pixel. By explicitly modeling this trust, the system can correct weak features using neighborhood context from the stronger modality and adaptively reweight their contributions during final classification.

What Problem Does This Paper Address?

In remote sensing, HSI provides rich spectral detail (what things are made of), while LiDAR provides elevation data (how tall things are). While complementary, both have "blind spots." HSI suffers from spectral similarity and mixed pixels, while LiDAR can’t distinguish between objects of similar heights (like a roof vs. a raised walkway).

Most existing fusion methods treat these modalities as equally reliable across an entire scene or use simple attention mechanisms to combine them. However, a modality that is highly informative in one region might be completely ambiguous in another. The authors argue that we shouldn't just fuse these features; we should use the "reliable" modality to correct the "unreliable" one before they are ever combined.

Methods

The proposed framework, ReDF (Reliability-Driven Framework), consists of two main stages following initial feature encoding:

  1. Reliability-Guided Feature Correction (RGFC):
    • Prototype Bank: The system maintains a "bank" of representative feature vectors (prototypes) for each land-cover class.
  2. Evidence Estimation: For every pixel, the model compares the local feature to these prototypes. High similarity equals high "evidence" and thus high reliability.
  3. Correction: The model creates a shared reference representation. If one modality has low reliability at a specific spot, it is "corrected" by pulling in neighborhood information from the other, more reliable modality.
  4. Reliability-Guided Feature Reweighting (RGFR):
    • Once features are corrected, the model needs to decide how much each should contribute to the final classification.
  5. Low-Rank Modulation: Using a technique called Canonical Polyadic (CP) decomposition, the model modulates the modality-specific features. This modulation is conditioned on a "reliability vector" that summarizes cross-modal disagreement and uncertainty (vacuity).
Fig. 1: Overall architecture of ReDF. (a) Modality encoders extract features from aligned HSI and LiDAR patches. (b) Class-prototype matching provides modality reliability estimates, while neighborhood agreement guides bidirectional cross- modal feature correction. (c) The modality features and reliability estimates are used to construct a reliability-weighted shared representation. The shared and modality-specific residual descriptors, together with the reliability information from (b), drive CP low-rank modulation to reweight the residual descriptors. The shared descriptor is preserved and concatenated with the reweighted residual descriptors for final classification.

Datasets

The authors evaluated ReDF on three standard HSI–LiDAR benchmarks:

  • Houston2013: An urban dataset from the University of Houston campus (15 classes).
  • MUUFL: Data from the University of Southern Mississippi’s Gulf Park campus, known for challenging mixed spatial contexts (11 classes).
  • Trento: A rural dataset from southern Italy with high spatial resolution (6 classes).

Key Insights and Ideas

  • Reliability as a Local Property: Reliability isn't a global setting; it varies pixel by pixel. By using class prototypes, the model gains a "semantic anchor" to judge how much it should trust a specific sensor at a specific location.
  • Correction Before Fusion: The "aha moment" here is that simply weighting features during fusion isn't enough. By using the reliable modality to guide a neighborhood-based correction of the weak modality, the model improves the discriminative power of the features before they are merged.
  • Efficiency through Low-Rank Decomposition: Handling the interaction between high-dimensional features and reliability statistics can be computationally expensive. The use of CP low-rank modulation (shown in Figure 2) allows the model to learn complex, conditioned reweighting without a massive parameter explosion.
Fig. 2: Reliability-Guided Feature Reweighting (RGFR). Conditioned on reliability information from the preceding RGFC stage, CP low-rank modulation reweights the modality-specific residual descriptors, with a learned scalar β controlling the modulation amplitude. The shared descriptor bypasses reweighting and is concatenated with the reweighted residuals for classification. S denotes stacking, C denotes concatenation, and filled-dot nodes denote element-wise multiplication.

Key Results

ReDF achieved state-of-the-art results across all three datasets, outperforming recent methods like ACFFN and mPMCL.

  • Houston2013: Reached an Overall Accuracy (OA) of 95.82%, notably improving performance on difficult classes like "Commercial" and "Stressed Grass."
  • MUUFL: Achieved 84.91% OA. This is a significant jump (over 1 percentage point) compared to the next best method, which is impressive given the complexity of this dataset.
  • Trento: Reached a near-perfect 99.56% OA.

Ablation studies confirmed that the Correction (RGFC) module is the most critical component, particularly in complex scenes where one sensor frequently fails to provide clear evidence.

Fig. 3: Comparative classification maps for the Houston2013 dataset. (a) Ground-truth map. (b) CoupledCNN (94.57%). (c) HCT (93.06%). (d) DSHFNet (90.23%). (e) HFSNet (90.35%). (f) DCU-Net (92.93%). (g) MSAF (92.89%). (h) ACFFN (95.50%). (i) mPMCL (94.31%). (j) ReDF (95.82%). Values in parentheses are the ten-run mean OA from the corresponding table, not the OA of the individual displayed map.

Limitations and Future Work

While ReDF performs excellently on aggregate, the authors note it still struggles with specific categories like "Sidewalk" and "Yellow Curb" in the MUUFL dataset. These categories often exist in highly mixed spatial contexts where a single pixel might contain multiple land-cover types, confusing the prototype-based evidence system.

Future work will likely focus on finer-grained spatial supervision and improving how the model handles patches containing multiple overlapping classes to better resolve these ambiguous boundaries.

Summary updated

Abstract

Hyperspectral image (HSI) and LiDAR data provide complementary spectral and elevation information for land-cover classification. However, the spatially varying class-discriminative reliability of modality features is rarely modeled explicitly, allowing weak or ambiguous features to adversely affect cross-modal interaction and fusion. To address this issue, we propose ReDF, a reliability-driven framework that quantifies the reliability of each modality's features at each spatial location through the strength of class-related evidence they provide, and uses it to guide feature correction and fusion. Specifically, a Reliability-Guided Feature Correction module introduces a class prototype bank for each modality and converts feature-prototype similarities into class evidence for reliability estimation. The estimated reliabilities are used to construct a reliability-weighted shared representation and to guide bidirectional cross-modal feature correction based on neighborhood agreement, refining modality features with weak class evidence through information from the other modality. Building on these outputs, a Reliability-Guided Feature Reweighting module conditions a low-rank modulation on the estimated reliabilities and the cross-modal disagreement, and applies it to the corrected modality-specific descriptors to adaptively rescale each modality's contribution to the fused representation used for final classification. Experiments on three public HSI-LiDAR datasets show higher mean overall accuracy and Kappa than the compared methods.
BibTeX
@misc{zhang2026reliability,
  author = {Lianming Zhang and Xiaolu Wang and Qiya Song and Jinyang Liu and Renwei Dian and Xudong Kang},
  title = {{Reliability-Driven Feature Correction for Hyperspectral and LiDAR Classification}},
  year = {2026},
  eprint = {2611.22508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22508},
}
Figure