A hyperspectral camera records a full reflectance spectrum at every pixel, often across a hundred or more narrow color bands. That spectrum can tell a painted car roof from asphalt when an ordinary photo cannot. Detectors trained on one such camera, however, rarely survive a move to another. Changing sensors changes two things at once. The bands shift, and the ground distance covered by one pixel (the ground sampling distance) grows or shrinks. These two changes are physically coupled. A car covering fifty clean pixels from a low drone becomes, from a high aircraft, a handful of mixed pixels whose spectra blend car and road, and eventually a single contaminated pixel. The same object therefore changes both its apparent scale and its spectral signature. Current methods treat these as separate problems: spectral sensor gaps for pixel classifiers, scale variation for RGB aerial detectors, and style shift for RGB domain adaptation.
We propose to treat the shift as one problem with a known forward model. Starting from a fine-resolution labeled source cube, we synthesize what a coarser, differently-banded target sensor would see. The simulation applies the target's optical blur (its point-spread function), averages pixels spatially so that materials mix linearly, and resamples bands through the target's spectral response curves. Boxes are converted into sub-pixel "abundance" labels once objects fall below a few pixels. A single detector is trained on the source and on these simulated sensors in parallel branches. The fine-resolution branch then teaches the coarse branches through their outputs (boxes, classes, abundance maps), not through their internal features. In two sentences: sensor transfer for hyperspectral detection is mostly a known physical degradation, so we should simulate it and distill across it rather than learn to ignore it adversarially. We evaluate that claim under a lab-to-field protocol, borrowed from machine diagnostics, that reports which axis of shift causes the error.
Core Claim
For hyperspectral object detection across sensors that differ in both bands and ground sampling distance, physically simulated target-sensor branches plus output-level cross-resolution distillation will close more of the source-to-target accuracy gap than feature alignment or camera-agnostic encoding, at equal compute. The gain should be concentrated on small and sub-pixel objects.
The claim is falsifiable at two points. The first is whether simulation-based training beats the adaptation baselines on a real held-out sensor. The second is whether distillation adds anything beyond simply augmenting the training data with the same simulations. We treat the second as the more dangerous question.
Where the Pieces Come From
Three bodies of work motivate the gap. In hyperspectral transfer, Hyper-CycleGAN translates spectra between sensors with an adversarial generator, and Tri-Former with Attention-Gated Tuning freezes a base model while tuning a small side branch for a new sensor. Both work on pixel classification at a fixed patch size and never model resolution change. CARL handles arbitrary band sets by encoding each channel with its physical wavelength, but it feeds a flat vision transformer for segmentation. It has no detection head and no multi-scale pyramid. We reuse CARL's wavelength encoding as an input layer. Unlike CARL, we do not expect the encoding alone to handle the resolution half of the shift.
In hyperspectral detection, S2ADet runs separate spectral and spatial streams through feature pyramids on the HOD3K benchmark. Its authors single out small, distant people as its main failure. SpecDETR detects sub-pixel point targets with a transformer. Its authors report degradation under shadow, canopy and new sensor noise when moving from simulated to real cubes, and they name cross-domain adaptation as future work. Both detectors are evaluated only in-domain. Each covers only one end of the size range: S2ADet handles resolved objects and SpecDETR handles points. Our detector must cover both, because a coarser sensor pushes resolved objects into the point regime.
In multi-scale and cross-domain RGB detection, MSCD supplies the core training mechanism. It trains shared-weight branches at 0.5x, 1.0x and 1.5x resolution and distills them into one inference branch. That work uses plain image rescaling on single-sensor RGB imagery. We replace the rescaling with physical sensor simulation, which in hyperspectral data changes the spectra as well as the size.
D-adapt constrains our design. It shows that aligning detector features across domains adversarially erodes the separation of foreground from background, and it adapts class and box predictions with separate modules instead. Finding Tiny Faces adds a second constraint: tiny objects are recognized from their surroundings rather than their interiors. Together these two results are why we distill outputs and not features. Forcing a coarse branch's features to match a fine branch's would demand exactly the scale invariance that tiny objects violate. It would also risk erasing the narrow absorption features that distinguish targets.
The diagnostic literature supplies the evaluation stance. Work on bearings and circuit breakers breaks lab-to-field failure down into operational, structural and severity shifts. That work shows near-perfect lab accuracy collapsing under combined shifts: one bearing network falls to about 73.9% under mixed noise at variable speed. It also shows that a physics forward model inside training can stabilize transfer, as in physics-informed transfer for circuit breakers. We adopt both ideas. We decompose shift into named axes, and we use a sensor forward model as the physical prior.
Research Program
The detector is a feature-pyramid network with an oriented-box head, as needed for aerial scenes where DOTA showed horizontal boxes fail on dense rotated objects. Its input layer is CARL-style wavelength-encoded, so any band set can enter. The pyramid's finest level adds a point head that predicts, per pixel, the fraction of that pixel covered by each class.
The sensor simulator takes a source cube and a target sensor description (spectral response curves, point-spread width, ground sampling distance ratio). Label transfer follows the geometry. A box that still spans more than about four target pixels stays a box. Below that, it becomes an abundance value computed from the exact area overlap. This is the regime where SpecDETR argues box regression becomes unstable.
Training runs the source branch and $K$ simulated-sensor branches with shared weights, following MSCD's structure. Each simulated branch receives two signals. The first is its own supervised loss on transferred labels. The second is a distillation loss toward the source branch's predictions, projected through the same forward model: boxes rescaled, and class maps area-averaged into abundance maps. That loss is gated, as in MSCD, by the teacher's own detection loss, so that confident teacher predictions dominate.
Finally, when unlabeled target imagery exists, we fit the simulator's free parameters by matching simple statistics of real target cubes. Those parameters are the point-spread width and a noise level. This step is self-calibration, not adversarial alignment.
Baselines
The naive reference is the source-only detector: the same architecture trained only on source data and applied directly to the target sensor after linear band interpolation. On HOD3K in-domain, S2ADet reports 59.8 mAP and a PCA-style Faster R-CNN reports 56.9 mAP (S2ADet). No reported cross-sensor result was found on Lacuna for either, so we measure both source-only numbers ourselves. For point targets, SpecDETR reports 0.856 mAP on the simulated SPOD benchmark and AP 1.0 on real SanDiego (SpecDETR). No reported result was found on Lacuna for SpecDETR on Avon or Gulfport, which we will run.
The strongest adaptation competitor is D-adapt applied to the same pyramid detector. It is the best-documented method for keeping detection quality under domain shift, reporting 49.0 mAP on VOC to Clipart1k against 27.8 without adaptation (D-adapt). No hyperspectral result for it was found on Lacuna. D-adapt gets the same unlabeled target images we use for self-calibration and the same number of GPU-hours.
The strongest sensor-invariance competitor is a CARL-pretrained encoder with our pyramid and head, fine-tuned on source only. It isolates whether wavelength encoding alone fixes the transfer. CARL reports 50.1 mIoU against 44.6 for camera-specific models on HSICity segmentation (CARL); no detection result was found on Lacuna.
The strongest scale competitor is MSCD with ordinary bicubic rescaling in place of physical simulation. It reports a 6.50 mAP gain over RoI Transformer on DOTA-v1.0 (MSCD). This baseline isolates whether physical degradation matters or whether any multi-resolution training suffices.
Two ablations isolate the key ingredient. The first trains on simulated-sensor data as plain augmentation, without distillation. The second distills intermediate features instead of outputs.
All methods share the backbone, training schedule, hyperparameter search budget (the same number of configurations tuned on a source-only validation split), and inference cost. Beating a baseline means a higher target-domain mAP averaged over three seeds with non-overlapping confidence intervals. In particular we require a margin on the small-object and point-target subsets, where the claim predicts the gain lives.
Why not just…?
Why not just reduce the cube to three principal components and fine-tune a strong RGB detector? On HOD3K this already loses in-domain, 56.9 versus 59.8 mAP (S2ADet). Principal components are also computed per sensor, so their meaning changes across sensors. A pretrained RGB detector has no prior for single-pixel targets that lack shape. We still include this pipeline, with D-adapt on top, as a cross-sensor baseline, because it is what practitioners would actually try.
Why not just align features adversarially, or translate target images to look like the source? D-adapt's own analysis argues global feature alignment trades away discriminability. Hyper-CycleGAN needed multi-scale attention and gradient-penalized training just to reach 90.02% OA on a classification transfer where plain CycleGAN reached 83.75%. Neither handles resolution change. A generator also cannot recover purity that coarser pixels have physically lost. The D-adapt baseline tests this directly. The forward model is only an advantage where it is correct, so we also run the simulator with deliberately wrong point-spread widths to measure how gracefully it degrades.
Why not just use camera-agnostic wavelength encoding and rely on multi-scale testing? CARL fixes the band mismatch but not the mixing of materials at coarse resolution. Multi-scale testing costs up to roughly seven times the inference time according to MSCD, and rescaling an image does not reproduce mixed spectra. The CARL baseline and the bicubic-MSCD baseline settle this.
Why not just augment with the simulated sensors and skip distillation? This is the real threat. If physical augmentation alone matches the full method, the honest conclusion is that a good forward model is the contribution and distillation is unnecessary. We report that outcome as such. Our prediction is that distillation helps specifically in the sub-pixel regime. There, transferred labels are only area-overlap estimates, whereas the teacher's predictions carry context from the fine-resolution view. The augmentation-only ablation, broken down by object size, decides it.
Proposed Experiments
The first experiment is diagnostic and fully controlled. Starting from HOD3K (16 bands, 470–620 nm), we generate target sensors that vary one axis at a time: bands only, resolution only, point-spread only, and all three together. This mirrors the operational, structural and severity decomposition in diagnostics. We report the fraction of the gap closed,
$$ G = \frac{m_{\text{method}} - m_{\text{source}}}{m_{\text{oracle}} - m_{\text{source}}}, $$
where $m$ is target mAP and "oracle" is the detector trained on labeled target data. This tells us which axis drives failure. It is circular for our method, since it tests on the simulator's own distribution, so we treat it as diagnosis only. Its value lies in the mismatched-simulator runs.
The decisive experiments use real target data. First, SPOD to real Avon, SanDiego and Gulfport point targets. This is SpecDETR's own simulated-to-real gap, and there we must match SpecDETR where it is saturated (SanDiego) and beat it on the scenes with canopy and shadow. Second, HOD3K to HOD-1, evaluating on the shared classes. We will verify the sensor metadata and class overlap before committing to this pair. Third, because real cross-sensor detection pairs are scarce, we will annotate vehicles in WHU-Hi drone scenes, captured with a Headwall sensor that differs in bands and resolution. That set is used only as a held-out field test and never for tuning.
Metrics are mAP, small-object and point-target AP, $G$, and inference latency. The proposal succeeds if the full method has the highest $G$ on real targets, beats augmentation-only on small and point objects, and holds up when the simulator is misspecified by a factor of two in point-spread width. It fails if CARL-plus-source-training or D-adapt matches it on the real pairs.
References
- Hyper-CycleGAN: A New Adversarial Neural Network Architecture for Cross-Domain Hyperspectral Data Generation
- Bridging Sensor Gaps via Attention-Gated Tuning for Hyperspectral Image Classification
- CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis
- Object Detection in Hyperspectral Image via Unified Spectral-Spatial Feature Aggregation
- SpecDETR: A transformer-based hyperspectral point object detection network
- Multi-Scale Cross Distillation for Object Detection in Aerial Images
- Decoupled Adaptation for Cross-Domain Object Detection
- Finding Tiny Faces
- Attention activation network for bearing fault diagnosis under various noise environments
- Fault diagnosis method for high-voltage circuit breakers based on physics-informed transfer learning
- DOTA: A Large-Scale Dataset for Object Detection in Aerial Images