Abstract
Advanced interpretation of hyperspectral remote sensing images benefits many precise Earth observation tasks. Recently, visual foundation models have promoted remote sensing interpretation but concentrated on RGB and multispectral images. Due to the varied hyperspectral channels, existing foundation models face image-by-image tuning situations, imposing great pressure on hardware and time resources. In this paper, we propose a tuning-free hyperspectral foundation model called HyperFree, by adapting existing visual prompt engineering. To process varied channel numbers, we design a learned weight dictionary covering the full-spectrum from $0.4 \sim 2.5 \, \mu\text{m}$, supporting dynamic construction of the embedding layer. To make prompt design more tractable, HyperFree generates multiple semantic-aware masks for one prompt by treating feature distance as semantic similarity. Pre-trained on a large-scale high-resolution hyperspectral dataset, HyperFree (1 prompt) achieves results comparable to specialized models (5 shots) across 5 tasks and 11 datasets.
Introduction
Hyperspectral imaging (HSI) captures hundreds of narrow spectral bands, providing a "high-definition" view of the Earth's surface. This spectral richness allows for the precise identification of materials, such as specific crop types or minerals. However, the diversity of HSI sensors—each capturing different wavelengths and band counts—presents a significant challenge for standard Remote Sensing Foundation Models (RSFMs).
Most existing RSFMs are designed for fixed-channel inputs (e.g., RGB). Consequently, applying these models to HSI typically requires a "Pre-training and Tuning" (P-T) paradigm, where the model must be fine-tuned for every new sensor or image. This process is computationally expensive and slow. Furthermore, while prompt-based models like the Segment Anything Model (SAM) have shown promise, they often struggle with the spectral complexity of HSI and typically generate only a single mask per prompt, whereas remote sensing tasks often require identifying all instances of a specific class.
Methodology
HyperFree addresses these challenges through two core innovations: a channel-adaptive embedding layer and a semantic-aware prompt interaction mechanism.
Channel-adaptive Embedding
To handle the varied number of channels across different HSI sensors, HyperFree utilizes a "learned weight dictionary" that covers the full spectrum from 0.4 to 2.5 $\mu$m. When an image is processed, the model dynamically constructs its embedding layer by retrieving weights corresponding to the specific wavelengths captured by the sensor. This approach makes the model inherently channel-agnostic and eliminates the need for per-sensor architectural adjustments.
Prompt-Mask-Feature (PMF) Interaction
HyperFree moves beyond simple binary segmentation by mapping prompts, generated masks, and backbone features into a shared semantic feature space. By treating the distance between these features as a measure of semantic similarity, the model can perform "reasoning." A single point prompt can thus trigger the identification of all semantically similar objects in a scene, effectively generating multiple masks from one prompt.
Hyper-Seg Data Engine
To overcome the scarcity of labeled HSI data, the authors developed the Hyper-Seg Data Engine. This automated pipeline leverages existing foundation models (e.g., SAM-H) to generate high-quality masks for unlabeled hyperspectral data. This process resulted in the Hyper-Seg dataset, containing nearly 50,000 HSI pairs and over 15 million masks.
Experimental Results
HyperFree was evaluated on 11 public datasets across five tasks: Multi-class Classification (HC), One-class Classification (HOCC), Target Detection (HTD), Anomaly Detection (HAD), and Change Detection (HCD).
Key findings include:
- State-of-the-Art Performance: In a tuning-free, 1-prompt setting, HyperFree achieved results comparable to specialized models trained with 5-shot supervision. Notably, it showed a 12-point improvement in F1 score for classification tasks over previous benchmarks.
- Efficiency: By bypassing the fine-tuning stage, HyperFree is 1 to 2 orders of magnitude faster than traditional RSFMs. For instance, it completed a classification task in approximately 7.85 seconds, compared to nearly 200 seconds for competing models.
- Robustness: The frozen backbone successfully handled diverse downstream tasks, demonstrating its versatility as a true foundation model for hyperspectral data.
Conclusion
HyperFree represents a significant step toward a "GPT moment" for hyperspectral remote sensing. By enabling tuning-free, channel-adaptive analysis, it makes high-precision Earth observation more accessible and efficient. Future work will focus on reducing dependency on precise wavelength metadata and improving global generalization by incorporating more diverse spaceborne hyperspectral data.
Summary updated