An End-to-End Joint Unsupervised Learning of Deep Model and Pseudo-Classes for Remote Sensing Scene Representation

Zhiqiang Gong, Ping Zhong, Weidong Hu, Fang Liu, Bingwei Hui

Abstract

This work develops a novel end-to-end deep unsupervised learning method based on convolutional neural network (CNN) with pseudo-classes for remote sensing scene representation. First, we introduce center points as the centers of the pseudo classes and the training samples can be allocated with pseudo labels based on the center points. Therefore, the CNN model, which is used to extract features from the scenes, can be trained supervised with the pseudo labels. Moreover, a pseudo-center loss is developed to decrease the variance between the samples and the corresponding pseudo center point. The pseudo-center loss is important since it can update both the center points with the training samples and the CNN model with the center points in the training process simultaneously. Finally, joint learning of the pseudo-center loss and the pseudo softmax loss which is formulated with the samples and the pseudo labels is developed for unsupervised remote sensing scene representation to obtain discriminative representations from the scenes. Experiments are conducted over two commonly used remote sensing scene datasets to validate the effectiveness of the proposed method and the experimental results show the superiority of the proposed method when compared with other state-of-the-art methods.

Introduction

High-resolution remote sensing imagery from satellites and aerial platforms is abundant, yet labeling these images for supervised learning is labor-intensive and requires significant domain expertise. Unsupervised learning offers a solution by extracting features without manual labels. However, remote sensing scenes present unique challenges: high intra-class variance (similar scenes looking different) and low inter-class variance (different scenes looking similar).

Existing unsupervised methods often rely on static, hand-crafted features or pre-defined pseudo-labels that do not evolve during training. This work proposes an end-to-end approach where the feature extractor and category definitions improve each other iteratively in a virtuous cycle.

Methodology

The proposed framework utilizes a Convolutional Neural Network (CNN) to learn discriminative representations through a joint learning process. The core innovation is the dynamic construction of pseudo-classes using center points in the feature space.

Overview of the proposed CNN model and joint learning loss framework.

Pseudo-Class Construction and Labeling

The model maintains a set of "center points" representing pseudo-classes. For each training batch, images are assigned pseudo-labels based on their proximity to these centers. This process allows the model to create its own supervision signal dynamically.

Illustration of the pseudo-labeling process where unlabeled features are assigned to pseudo-classes based on the nearest center points.

Joint Loss Function

The network is optimized using a combination of two losses:

  1. Pseudo-Softmax Loss ($L_s$): This treats the generated pseudo-labels as ground truth to train the CNN to discriminate between clusters.
  2. Pseudo-Center Loss ($L_c$): This minimizes the distance between a sample's features and its assigned center point. Crucially, this loss updates both the CNN parameters and the positions of the center points themselves via backpropagation.

This joint optimization allows the "moving target" of cluster centers to refine alongside the feature extractor, preventing the model from stagnating on poor initial clusters and ensuring the features remain discriminative.

Experimental Evaluation

The method was validated on two widely recognized benchmarks: the UC Merced Land Use Dataset (21 classes) and the Brazilian Coffee Scene Dataset (2 classes).

Key Results

The proposed method demonstrated significant improvements over both traditional hand-crafted methods and other deep unsupervised approaches:

  • UC Merced: Achieved 94.33% accuracy, outperforming Dense SIFT (81.67%), MARTA GANs (85.37%), and UCFFN (88.57%).
  • Brazilian Coffee: Achieved 87.74% accuracy, surpassing standard CNN-1 models (86.00%) and SIFT (82.83%).

Analysis revealed that the tradeoff parameter $\lambda$ (balancing the two losses) is critical; the model performs best when there is a balance between collapsing samples into centers and maintaining discriminative power. Furthermore, the model often performed best when the number of pseudo-classes was smaller than the actual number of ground-truth classes, suggesting it effectively learns broad semantic groupings first.

Conclusion

This work demonstrates a powerful end-to-end unsupervised learning framework for remote sensing scene representation. By jointly optimizing feature extraction and cluster center refinement, the model achieves state-of-the-art accuracy without human labels. Future research will explore more advanced architectures like ResNet and the application of this joint learning framework to other complex tasks such as object detection.

Summary updated

Abstract

This work develops a novel end-to-end deep unsupervised learning method based on convolutional neural network (CNN) with pseudo-classes for remote sensing scene representation. First, we introduce center points as the centers of the pseudo classes and the training samples can be allocated with pseudo labels based on the center points. Therefore, the CNN model, which is used to extract features from the scenes, can be trained supervised with the pseudo labels. Moreover, a pseudo-center loss is developed to decrease the variance between the samples and the corresponding pseudo center point. The pseudo-center loss is important since it can update both the center points with the training samples and the CNN model with the center points in the training process simultaneously. Finally, joint learning of the pseudo-center loss and the pseudo softmax loss which is formulated with the samples and the pseudo labels is developed for unsupervised remote sensing scene representation to obtain discriminative representations from the scenes. Experiments are conducted over two commonly used remote sensing scene datasets to validate the effectiveness of the proposed method and the experimental results show the superiority of the proposed method when compared with other state-of-the-art methods.
BibTeX
@misc{gong2019end,
  author = {Zhiqiang Gong and Ping Zhong and Weidong Hu and Fang Liu and Bingwei Hui},
  title = {{An End-to-End Joint Unsupervised Learning of Deep Model and Pseudo-Classes for Remote Sensing Scene Representation}},
  year = {2019},
  eprint = {1903.07224},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.07224},
}
Figure