Confusion-Targeted Band Selection for Label-Scarce Hyperspectral Classification

A hyperspectral image records, for every pixel, reflectance in a hundred or more narrow wavelength bands. With few labeled pixels, classifiers fitted on all those bands overfit. This is the Hughes phenomenon, a form of the curse of dimensionality: accuracy falls as the band count grows while the label count stays fixed. The usual remedy is to choose or compress bands once for the whole image. A single global choice is a poor fit for a scene made of many materials, though. SuperPCA showed that running PCA separately inside spatially homogeneous regions raises Indian Pines overall accuracy from 67.27% (global PCA) to 96.78%.

We propose a different kind of localization, driven by where the classifier is unsure rather than by geometry. A cheap global model with a small, sparse set of bands handles every pixel. Where that model is uncertain, its two top-scoring classes tell us which confusion to resolve. For that class pair only, we fit a second sparse band selector that is free to pick bands the global model discarded. Per-band evidence is encoded the way biologists score sequence positions: as a position-specific scoring matrix (PSSM), a table of log-odds scores for each "position" (here, each band) with pseudocounts that keep estimates stable when examples are scarce. The prediction is that most hard pixels are hard because of a few class pairs that are separated by bands which are rare in the global support. A small number of pairwise sparse refits should then recover much of the gap to heavy models in the low-label regime, using a fraction of the bands and parameters.

Core Claim

The bands that separate two specific confusable classes differ from the bands that best separate all classes on average. Global L1 selection (a penalty on the absolute size of coefficients that drives most of them to exactly zero) therefore systematically discards bands that matter for a few confusions. Uncertainty from the global model is a cheap, label-free signal that points to exactly those confusions.

The novelty is the combination of three elements. Refinement is triggered by uncertainty. The refinement acts along the spectral axis, by re-selecting bands. The re-selection is conditioned on the confused class pair rather than on spatial location.

Prior uncertainty-guided refinement only adapts spatial computation. FineSAM++ routes uncertain patches to small residual experts on 1–3 channel vessel images. Qin et al. route uncertain patches to full resolution. Hyperspectral sparsity methods, on the other hand, are static. SpecTr applies one sparse attention constraint (α-entmax, which can assign exactly zero weight to bands) uniformly over every location. CUR band selection picks one unsupervised band subset. Online sparse streaming selection with uncertainty uses uncertainty to decide whether features enter a single global model, but only on tabular streams. None of these changes the selected spectral support per confusion.

Research Program

Stage 1: spectral scoring matrix. Each band $b$ is quantized into $K$ bins. The score for class $c$ at band $b$ and bin $k$ is

$$s_{c,b}(k)=\log\frac{(n_{c,b,k}+\alpha\,p_{b,k})/(N_c+\alpha)}{p_{b,k}},$$

with the following definitions. $n_{c,b,k}$ is the number of labeled class-$c$ pixels whose band-$b$ value falls in bin $k$. $N_c$ is the number of labeled class-$c$ pixels. $p_{b,k}$ is the background frequency of bin $k$, estimated from all pixels, including unlabeled ones. $\alpha$ is a pseudocount weight.

This is exactly the construction of a sequence PSSM, with bands in place of residue positions. We should be candid that summing such scores is a discretized naive Bayes classifier. What the PSSM view adds is three things:

  • a principled background model that uses unlabeled pixels,
  • pseudocount smoothing designed for shallow data, the analogue of shallow homology,
  • a per-band "conservation" measure. This is the relative entropy between two class profiles at that band, which we use as a prior on band relevance.

W-PSSKM supplies a further tool. It showed that position-specific profiles fed through a positive semi-definite product kernel beat ESM-2 on Spike7k (0.861 vs 0.480). We reuse its kernel form as an optional SVM over band-score vectors. Its role is to provide method, not to motivate the work. The PSSM encoding stays only if it beats a Gaussian-likelihood encoding of the same bands, and we will drop it if not.

Stage 2: global sparse model. A multiclass L1 logistic regression on the score features yields a small global band support. Lasso shrinks the coefficients it keeps, so we then refit on that support without the L1 penalty. This relaxed-lasso debiasing is borrowed from Sparse Relaxed-Lasso Steering. The predictive entropy and the top-two margin of this model form an uncertainty map.

Stage 3: confusion-pair refinement. Pixels above an uncertainty threshold are grouped by their top-two class pair $(c,c')$. For each pair that occurs often enough, we fit a binary sparse logistic model on the pairwise score differences $s_{c,b}-s_{c',b}$, using all $B$ bands, with penalty $\lambda\sum_{b}|\beta_b|/I_{cc'}(b)$. Here $\beta_b$ is the coefficient on band $b$, $\lambda$ sets the overall sparsity, and $I_{cc'}(b)$ is the relative entropy between the two classes' smoothed profiles at band $b$. The penalty is therefore weakest on bands that best discriminate this particular pair.

The pairwise model is trained only on labeled pixels of $c$ and $c'$. This is far fewer parameters than a local model per superpixel, which is how the method avoids the overfitting that per-region Lasso would suffer. The model is then debiased with the same unrestricted refit on its support. Its output replaces the global decision only for the routed pixels.

Analysis. We will measure three things:

  • the overlap between pairwise and global supports,
  • whether pair-specific bands correspond to known absorption features, for example in the choledoch pathology data,
  • how accuracy on routed pixels varies with the routing fraction.

We will also check calibration explicitly. Feng et al. warn that refinement confined to uncertain regions inherits any miscalibration of the uncertainty estimate.

Baselines

Naive reference. A linear SVM and an L2 logistic regression on all raw bands. This shows the size of the Hughes penalty at each label budget.

Global PCA plus SVM. Its reported Indian Pines OA is 67.27% (SuperPCA).

Strongest classical competitor: multiscale SuperPCA with SVM voting. Its reported Indian Pines OA is 96.78% (SuperPCA). It is the closest rival because it is the existing answer to "global reduction fails, so localize."

Strongest deep competitor on remote sensing: SCT-Net. Its reported OA is 98.67% on Salinas and 97.72% on Houston2018 (SCT-Net).

Pathology competitor: SpecTr. Its reported DSC on the choledoch dataset is 75.21% (SpecTr).

Fair comparison. Published numbers come from those papers' own splits, and the file gives no reported results at the label budgets we target. We will therefore rerun MSuperPCA, SCT-Net, and global relaxed lasso ourselves. All methods get identical training pixels at 5, 10, and 20 labels per class and at the published splits, with the same hyperparameter-search budget measured in configurations tried. Each method's spatial post-processing will be either kept or removed for all methods.

What counts as success. We claim success if the method matches or exceeds rerun MSuperPCA OA at 5–10 labels per class while using at most a fifth of the bands per decision. We also want it to come within one OA point of rerun SCT-Net at ten or more times lower training compute. If it only matches global relaxed lasso, the idea fails.

Ablations.

  • Routing test. Route a random set of pixels of the same size instead of the uncertain ones. This tests whether uncertainty matters.
  • Pairing test. Refit by spatial superpixel instead of by confusion pair. This tests whether conditioning on the pair matters.
  • Encoding test. Replace the PSSM scores with Gaussian log-likelihoods.
  • Penalty test. Remove the relative-entropy penalty weights.
  • Debiasing test. Remove the debiasing refit.

Why not just…?

Why not just use SuperPCA, which already localizes? SuperPCA localizes by spatial homogeneity, without supervision. It cannot know that two adjacent crop types are confused, and its accuracy depends on how well the superpixels follow true boundaries. That dependence on segmentation is its stated weakness. This is a real threat at standard label budgets. The superpixel-refit ablation and the head-to-head at 5 labels per class will settle it. If superpixel conditioning matches pair conditioning, the claim about spectral confusion is wrong, and we will report it as wrong.

Why not just train a bigger deep network with dropout-based uncertainty? With ample labels, SCT-Net-style models are likely to win: 98.67% Salinas OA leaves little headroom. Our claim is restricted to the label-scarce regime, where dense spectral networks face the Hughes problem most directly, and to models whose decisions are interpretable because they use few bands. The label-budget sweep shows where the curves cross. If they cross at or below 5 labels per class, the proposal has little practical value.

Why not just a global Lasso with a larger band budget? If confusion-specific bands could simply be added globally, a looser global λ would capture them. That would cost estimation variance on every other class, though. We test this directly by comparing global relaxed lasso at the same total number of distinct selected bands as our global support plus all pairwise supports.

Why not just call this naive Bayes? Stage 1 is a smoothed, discretized naive Bayes, and we say so. The Gaussian-encoding ablation decides whether the PSSM machinery of background frequencies from unlabeled pixels and pseudocounts adds anything. If it does not, the method still stands on the routed pairwise sparse refits, with the encoding swapped out.

Proposed Experiments

Benchmarks. Indian Pines, Salinas, Pavia University, and Houston2018 for land cover. The 60-band choledoch pathology dataset tests transfer to a biological segmentation setting. There, routing operates on per-pixel cancerous-versus-healthy uncertainty, and the evaluation uses DSC and Hausdorff distance.

Metrics. Overall accuracy, Kappa, and per-class accuracy, plus:

  • accuracy restricted to routed pixels,
  • mean number of bands used per decision,
  • training and inference time,
  • expected calibration error of the global model's uncertainty.

First decisive check: routing beats chance. On routed pixels, the refit must lower error by much more than a random routing of the same size does. If not, uncertainty is not identifying fixable confusions.

Second decisive check: pair-specific bands exist. Pair-specific supports should overlap the global support only partially, for example a Jaccard overlap below 0.5. Removing the pairwise-only bands should erase the gain. This would show the improvement comes from bands that global selection discards, not from extra capacity.

Third decisive check: refinement depends on calibration. We will plot refinement gain against calibration error across datasets and label budgets. If gain disappears whenever calibration is poor, the method needs a calibrated uncertainty estimate, and we will treat that as a limitation rather than hide it.