Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

Thomas Snyder, H. Lexie Yang, Stefan Schnake, Steffen Schotthöfer

Abstract

Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption. In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer-based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.

Introduction

Geospatial Foundation Models (FMs), particularly those based on Vision Transformers (ViTs), have revolutionized remote sensing tasks such as land-cover mapping and disaster response. These models are typically pretrained on massive datasets and then fine-tuned for specialized downstream applications. However, their immense scale—often exceeding hundreds of millions of parameters—presents a significant barrier to deployment on edge hardware like drones or satellites, where memory and power are strictly limited.

Traditional compression techniques often face a stark trade-off between model size and accuracy. While Low-Rank Adaptation (LoRA) has gained popularity for its efficiency, it can suffer from significant performance degradation at high compression ratios. This research addresses the challenge of compressing large-scale ViTs during the transfer learning process without sacrificing the specialized performance required for complex geospatial data.

Methodology: Dynamical Low-Rank Training (DLRT)

The core of this approach is Dynamical Low-Rank Training (DLRT). Unlike standard training which updates full-rank weight matrices, DLRT constrains updates to a low-dimensional manifold by representing weight matrices as products of smaller matrices ($W = USV^\top$).

Conceptual illustration of manifold-constrained optimization in DLRT.

The DLRT framework operates through a dynamic three-step process:

  1. Basis Augmentation: The model's rank is temporarily doubled to explore optimal directions for learning from new geospatial data.
  2. Latent Space Training: Learning occurs within a projected, low-dimensional subspace, significantly reducing computational overhead.
  3. Truncation: Singular Value Decomposition (SVD) is employed to prune less significant information, retracting the model back onto the low-rank manifold.

A key advantage of DLRT over static methods like LoRA is that the basis matrices ($U$ and $V$) evolve during training. This flexibility allows the model to better align its internal structure with the specific requirements of the downstream geospatial task.

Experimental Setup

The researchers evaluated DLRT across three prominent remote sensing benchmarks:

  • UCM (UC Merced Land Use): 2,100 high-resolution images across 21 classes.
  • AID (Aerial Image Dataset): 10,000 images with 30 classes.
  • NWPU-RESISC45: A large-scale dataset of 31,500 images covering 45 categories.

Experiments utilized ViT models pretrained on ImageNet-21k and OReole-MR (a specialized geospatial foundation model).

Results and Key Insights

DLRT consistently demonstrated superior performance compared to both full-rank baselines and LoRA across all tested datasets and architectures.

Comparative performance of DLRT against baseline and LoRA across geospatial benchmarks.

Key Findings:

  • High Compression Ratios: The method achieved parameter reductions between 64% and 82%.
  • Minimal Accuracy Loss: On the UCM dataset, DLRT maintained accuracy within 0.5–1.0% of the original uncompressed model.
  • Superiority over LoRA: DLRT outperformed LoRA by 1% to 6% in validation accuracy while achieving comparable or superior compression.
  • Pretraining Influence: Models pretrained with supervised learning (ImageNet) were found to be more amenable to low-rank compression than self-supervised models (OReole), suggesting that supervised weights may naturally possess a more low-rank compatible structure.
  • Optimization Stability: DLRT proved robust to hyperparameter variations, showing less sensitivity to specific settings than standard full-rank training.

Conclusion and Future Work

This work demonstrates that manifold-constrained optimization via DLRT is a highly effective strategy for deploying large geospatial foundation models on edge devices. By dynamically adapting the model's low-rank structure during transfer learning, DLRT achieves significant compression with negligible performance loss.

Future research will explore making self-supervised pretraining more conducive to low-rank structures and extending this framework to more complex geospatial tasks such as semantic segmentation and object detection.

Summary updated

Abstract

Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption. In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer-based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.
BibTeX
@misc{snyder2026compressing,
  author = {Thomas Snyder and H. Lexie Yang and Stefan Schnake and Steffen Schotthöfer},
  title = {{Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization}},
  year = {2026},
  eprint = {2601.08882},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.08882},
}
Figure