Vision Transformer With Contrastive Learning for Remote Sensing Image Scene Classification

Meiqiao Bi, Minghua Wang, Zhi Li, Danfeng Hong

TL;DR

Researchers have developed ViT-CL, a framework that combines the global "big picture" capabilities of Vision Transformers (ViT) with the discriminative power of Supervised Contrastive Learning (SupCon). By using a novel two-stage optimization process, the model achieves state-of-the-art accuracy in classifying complex remote sensing scenes, effectively handling the high intraclass diversity and interclass similarity common in satellite imagery.

What Problem Does This Paper Address?

Remote Sensing Image (RSI) scene classification is notoriously difficult. Unlike standard computer vision tasks (like identifying a dog in a photo), a "school" scene in a satellite image might contain a baseball field, a parking lot, and several buildings scattered across a large area.

Traditional Convolutional Neural Networks (CNNs) excel at capturing local textures but often struggle with these long-range spatial relationships. Vision Transformers (ViTs) are better at capturing global context, but they have two major weaknesses:

  1. Data Hunger: They lack the "inductive biases" (like locality) of CNNs and usually require massive datasets to generalize well.
  2. Loss Function Limitations: Standard Cross-Entropy (CE) loss often fails to create distinct boundaries between classes that look very similar (e.g., a "freeway" vs. a "runway") or to group together classes that look very different (e.g., different types of "airports").

Methods

The authors propose ViT-CL, which stands for Vision Transformer with Contrastive Learning. The architecture consists of three main pillars:

  • The Backbone: Instead of training from scratch, they use a ViT-Base (B/16) model pretrained on ImageNet-21k. This provides the model with a strong starting point, overcoming the data-scarcity issue.
  • Supervised Contrastive (SupCon) Loss: Unlike standard contrastive learning (which is often self-supervised), SupCon uses label information to pull all samples from the same class together in the embedding space while pushing samples from different classes apart. This "hard positive/negative mining" makes the features much more discriminative.
  • Two-Stage Optimization: ViTs are notoriously sensitive to hyperparameters. To stabilize training, the authors use a two-step approach:
  • Stage 1: Fine-tune the pretrained ViT using only standard Cross-Entropy loss to get a stable baseline.
  • Stage 2: Fine-tune the model further using a joint loss function (CE + SupCon). This allows the model to refine its feature clusters without losing the classification accuracy gained in the first stage.

Datasets

The model was evaluated on three benchmark remote sensing datasets:

  • AID (Aerial Image Dataset): 10,000 images across 30 categories.
  • NWPU-RESISC45: A massive, challenging dataset with 31,500 images and 45 categories.
  • UCM (UC-Merced Land Use Dataset): 2,100 images across 21 categories.

Key Insights and Ideas

  • Global Context is King: The paper reinforces that for remote sensing, the ability to link a "parking lot" on one side of an image to a "terminal building" on the other is vital for identifying an "Airport" scene—a task where ViT's self-attention mechanism shines.
  • The Power of "Pull and Push": By visualizing the feature space using t-SNE, the authors showed that SupCon loss creates much tighter, more distinct clusters than CE loss alone. This is the "aha moment": the model isn't just memorizing labels; it's learning a more logical mathematical representation of what makes a "forest" different from a "meadow."
  • Stability through Sequencing: The two-stage training strategy is a practical "engineering win." It suggests that for complex models like ViTs, jumping straight into multi-objective loss functions can be counterproductive; a "warm-up" phase with simpler objectives leads to better final convergence.

Key Results

ViT-CL outperformed several classic CNNs (like ResNet-50 and VGGNet-16) and other specialized ViT variants. The quantitative highlights include:

  • AID: Achieved 97.42% accuracy (at a 50% training ratio).
  • NWPU-RESISC45: Achieved 94.54% accuracy (at a 20% training ratio), which is particularly impressive given the dataset's complexity.
  • UCM: Achieved a near-perfect 99.76% accuracy.

Qualitatively, the model showed a remarkable ability to distinguish between "high interclass similarity" pairs, such as correctly identifying stadiums vs. playgrounds, which often look nearly identical from a bird's-eye view.

Limitations and Future Work

While highly effective, the authors note a few areas for improvement:

  • Computational Cost: Transformers are more computationally expensive than many CNNs, which might limit deployment in real-time or edge-computing scenarios.
  • Hyperparameter Sensitivity: Even with the two-stage approach, tuning the tradeoff parameter ($\lambda$) between the two loss functions is time-consuming.
  • Future Directions: The authors suggest exploring unsupervised contrastive learning or more advanced data augmentation strategies to further reduce the reliance on labeled data, which is expensive to produce for satellite imagery.

Summary updated

Abstract

Remote sensing images (RSIs) are characterized by complex spatial layouts and ground object structures. ViT can be a good choice for scene classification owing to the ability to capture long-range interactive information between patches of input images. However, due to the lack of some inductive biases inherent to CNNs, such as locality and translation equivariance, ViT cannot generalize well when trained on insufficient amounts of data. Compared with training ViT from scratch, transferring a large-scale pretrained one is more cost-efficient with better performance even when the target data are small scale. In addition, the cross-entropy (CE) loss is frequently utilized in scene classification yet has low robustness to noise labels and poor generalization performances for different scenes. In this article, a ViT-based model in combination with supervised contrastive learning (CL) is proposed, named ViT-CL. For CL, supervised contrastive (SupCon) loss, which is developed by extending the self-supervised contrastive approach to the fully supervised setting, can explore the label information of RSIs in embedding space and improve the robustness to common image corruption. In ViT-CL, a joint loss function that combines CE loss and SupCon loss is developed to prompt the model to learn more discriminative features. Also, a two-stage optimization framework is introduced to enhance the controllability of the optimization process of the ViT-CL model. Extensive experiments on the AID, NWPU-RESISC45, and UCM datasets verified the superior performance of ViT-CL, with the highest accuracies of 97.42%, 94.54%, and 99.76% among all competing methods, respectively.
BibTeX
@article{bi2022vision,
  author = {Meiqiao Bi and Minghua Wang and Zhi Li and Danfeng Hong},
  title = {{Vision Transformer With Contrastive Learning for Remote Sensing Image Scene Classification}},
  journal = {IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},
  year = {2022},
  doi = {10.1109/jstars.2022.3230835},
}
Figure