Online Sensor Hallucination via Knowledge Distillation for Multimodal Image Classification

Saurabh Kumar, Biplab Banerjee, Subhasis ChaudhuriarXiv 2019 · arXiv

Abstract

In remote sensing, multimodal data fusion—such as combining high-resolution panchromatic (PAN) images with spectral-rich multi-spectral (MS) data—is essential for accurate scene classification. However, real-world operational constraints often lead to missing modalities at inference time due to sensor failures or orbital gaps. This paper presents a novel framework for "online sensor hallucination" using knowledge distillation. By training a hallucination network (Hall-Net) to approximate the features of a missing modality from available ones, the system can leverage multimodal benefits even in a unimodal test environment. Experimental results on the PAN-MS and Pavia University datasets demonstrate that the proposed method significantly narrows the performance gap caused by missing sensors, achieving classification accuracies close to those of full multimodal systems.

Introduction

Multimodal fusion is a cornerstone of modern satellite image analysis. Different sensors provide complementary information: panchromatic (PAN) sensors capture high spatial detail, while multi-spectral (MS) sensors provide rich spectral signatures. While combining these streams yields superior classification performance, standard multimodal models are inherently brittle, requiring all input modalities to be present during both training and inference.

In practical scenarios, such as rapid disaster response or environmental monitoring, temporal gaps or hardware limitations often mean that only a subset of sensors is available. The challenge lies in developing a "robust" classifier that can maintain high accuracy when forced to operate in a unimodal environment at test time. This work addresses this by introducing a hallucination-based approach rooted in the "Learning Using Privileged Information" (LUPI) paradigm.

Methodology: Modality Hallucination

The core of the proposed framework is the Hallucination Network (Hall-Net), which learns to predict the discriminative features of a missing modality based on the available one. The training process is structured into four modular steps:

  1. Individual Stream Training: Separate convolutional neural networks (PAN-Net and MS-Net) are trained independently for each modality using standard cross-entropy loss ($L_{CE}$).
  2. Teacher Network Training: The individual streams are integrated into a joint "two-stream" network. A late-fusion strategy is employed to combine the features, and the entire network is trained to serve as a high-performance "teacher."
  3. Student Hallucination Training: The Hall-Net is introduced to take the available modality (e.g., PAN) as input. Its goal is to approximate the feature representation that the missing modality (e.g., MS) would have generated.
  4. Knowledge Distillation (KD): To ensure the hallucination is effective for classification, the Hall-Net is trained using knowledge distillation. Instead of matching raw pixels, it attempts to match the "soft" outputs (logits) of the teacher network. This process transfers the "dark knowledge"—the nuanced class relationships and inter-modal dependencies learned by the teacher—to the student.
The four-step training and inference methodology: (a) independent training, (b) joint teacher training, (c) student hallucination training via distillation, and (d) final inference with hallucinated features.

Experimental Evaluation

The authors evaluated the framework on two distinct remote sensing datasets:

  • PAN-MS Dataset: This dataset consists of 80,000 image pairs from the GF-1 satellite, categorized into eight land-cover classes, including forest, farmland, and urban areas.
  • Pavia University Dataset: A benchmark hyperspectral dataset. To simulate missing modalities, the 103 spectral bands were partitioned into two subsets, with the model tasked to hallucinate one subset from the other.

Results and Discussion

The experimental results confirm that hallucination significantly outperforms unimodal baselines. On the PAN-MS dataset, the full two-stream teacher network achieved a top accuracy of 98.23%. When restricted to only the PAN sensor without hallucination, accuracy fell to 95.82%. However, by employing the Hall-Net to hallucinate MS features from the PAN input, the accuracy was restored to 96.78%.

Key Insights

  • Feature vs. Pixel Reconstruction: The framework prioritizes approximating discriminative features over pixel-level reconstruction. This focus ensures that the hallucinated data is optimized specifically for the classification task.
  • The Role of Soft Targets: By training on the teacher's logits rather than hard labels, the Hall-Net learns how the model perceives similarities between classes. This provides a much richer training signal, especially when the available modality lacks the spectral depth of the missing one.
  • Temperature Tuning: The "Temperature" ($T$) parameter in the softmax function was found to be critical. Higher temperatures soften the probability distribution, facilitating the transfer of complex relationships. The authors identified $T=15$ as optimal for the MS hallucination task.

Conclusion

This paper introduces a robust framework for multimodal image classification under sensor constraints. By leveraging knowledge distillation to "hallucinate" missing sensor data, the proposed model maintains high performance even when critical data streams are unavailable at inference time. Future research will focus on scaling the framework to handle a larger number of heterogeneous sensors and incorporating temporal dynamics to address missing data in multi-temporal sequences.

Summary updated

Abstract

We deal with the problem of information fusion driven satellite image/scene classification and propose a generic hallucination architecture considering that all the available sensor information are present during training while some of the image modalities may be absent while testing. It is well-known that different sensors are capable of capturing complementary information for a given geographical area and a classification module incorporating information from all the sources are expected to produce an improved performance as compared to considering only a subset of the modalities. However, the classical classifier systems inherently require all the features used to train the module to be present for the test instances as well, which may not always be possible for typical remote sensing applications (say, disaster management). As a remedy, we provide a robust solution in terms of a hallucination module that can approximate the missing modalities from the available ones during the decision-making stage. In order to ensure better knowledge transfer during modality hallucination, we explicitly incorporate concepts of knowledge distillation for the purpose of exploring the privileged (side) information in our framework and subsequently introduce an intuitive modular training approach. The proposed network is evaluated extensively on a large-scale corpus of PAN-MS image pairs (scene recognition) as well as on a benchmark hyperspectral image dataset (image classification) where we follow different experimental scenarios and find that the proposed hallucination based module indeed is capable of capturing the multi-source information, albeit the explicit absence of some of the sensor information, and aid in improved scene characterization.
BibTeX
@misc{kumar2019online,
  author = {Saurabh Kumar and Biplab Banerjee and Subhasis Chaudhuri},
  title = {{Online Sensor Hallucination via Knowledge Distillation for Multimodal Image Classification}},
  year = {2019},
  eprint = {1908.10559},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.10559},
}
Figure