RS-JEPA: Remote Sensing Joint-Embedding Predictive Architecture with Scan-Path Views for Cross-Sensor Image Retrieval

Tanishq Rachamalla, Srishti Kaushik, Muhammad Attique Khan, Gustau Camps-Valls, Swalpa Kumar RoyarXiv 2026 · arXiv

TL;DR

RS-JEPA is a compact, self-supervised framework for cross-sensor remote sensing image retrieval that leverages the unique sequence-dependent properties of Vision Mamba. By treating different image reading orders as "scan-path views" and using adversarial training to prevent shortcuts, it achieves state-of-the-art accuracy with a retrieval network of just 3.88M parameters—up to 16x smaller than previous transformer-based architectures.

What Problem Does This Paper Address?

In remote sensing, we often need to find the same scene across different sensor types—for example, matching a cloudy Synthetic Aperture Radar (SAR) image to a clear-sky optical image. This is a "cross-sensor retrieval" problem.

It is notoriously difficult for three reasons:

  1. Heterogeneity: The same physical scene looks radically different across sensors (e.g., SAR speckle vs. optical color).
  2. Label Scarcity: Manually labeling millions of satellite images is impossible, necessitating self-supervised learning.
  3. Efficiency: Large-scale archives require small, fast encoders. Existing methods often rely on massive Vision Transformers (ViTs) that are computationally expensive to run over entire archives.

The authors identify a gap: while Joint-Embedding Predictive Architectures (JEPAs) have shown promise by predicting features instead of pixels (avoiding the trap of learning sensor-specific noise), they have remained bulky.

Methods

The core innovation of RS-JEPA lies in its use of Vision Mamba (Vim), a state-space model (SSM). Unlike Transformers, which are order-agnostic due to global attention, SSMs process images as ordered sequences. This "architectural bias" becomes a feature, not a bug.

1. Scan-Path Views

Because a Mamba encoder's output for a specific patch depends on the tokens it has already "read," the authors propose using different traversal orders (e.g., row-major vs. column-major) of the same image as distinct views. This provides a natural source of self-supervision without needing complex image augmentations or momentum encoders.

2. Adversarial View Coupling (AVC)

To prevent the model from simply interpolating nearby pixels (a "trivial" task), the authors introduced Adversarial View Coupling. A scorer learns to split the visible patches into two halves that are the hardest to use for predicting hidden patches. This forces the model to learn global scene context rather than local spatial continuity.

Figure 4. Adversarial View Coupling. A scorer with gradient reversal splits the visible tokens into views A and B, which the shared encoder reads along g1. In each of four CVA blocks, one view’s queries attend to the other view’s keys and values, giving XAB and XBA for the predictor.

3. Neighborhood Consensus Loss

Beyond aligning a SAR image with its exact optical twin, RS-JEPA uses a consensus loss to ensure that the "neighborhood geometry" is preserved. If two scenes are similar in the SAR domain, their embeddings should also be similar in the optical domain.

Overview of the RS-JEPA training and inference pipelines.

Datasets

The model was evaluated on three major dual-sensor benchmarks:

  • BEN-14K: A subset of BigEarthNet featuring Sentinel-1 (SAR) and Sentinel-2 (Optical) pairs.
  • CBRSIR-VS: SAR and RGB image pairs.
  • DSRSID: High-resolution panchromatic (PAN) and lower-resolution multispectral (MS) images.

Key Insights and Ideas

  • Reading Order as Supervision: The most elegant "aha moment" is realizing that for sequence-based models like Mamba, the path taken through the pixels is a valid self-supervised signal.
  • Feature Prediction > Pixel Reconstruction: By predicting latent features (JEPA style) rather than raw pixels (MAE style), the model ignores sensor-specific "noise" like SAR speckle, focusing instead on the underlying semantic content.
  • Small is Beautiful: The paper demonstrates that a carefully designed 3.88M parameter model can outperform 100M+ parameter models by better leveraging architectural priors.

Key Results

RS-JEPA achieved the highest F1@5 scores across all three benchmarks, outperforming twelve other re-implemented state-of-the-art methods.

  • Accuracy: It reached 72.92 (BEN-14K), 80.35 (CBRSIR-VS), and 94.43 (DSRSID) in mean F1@5.
  • Efficiency: The retrieval network uses only 3.88M parameters, which is 8–16× fewer than prior JEPA-based retrievers.
  • Throughput: As shown in Figure 1, RS-JEPA sits in the "sweet spot" of the Pareto front—offering the best retrieval accuracy while maintaining high throughput.
Figure 1. Accuracy against model size. Mean F1@5 over the three benchmarks against retrieval throughput (marker area: pa- rameters).

Limitations and Future Work

While impressive, the authors note a few limitations:

  • Co-registration Requirement: The model still requires co-registered image pairs for training, which can be a bottleneck for scaling to unaligned datasets.
  • Fixed Resolution: Experiments were conducted at a fixed 120x120 resolution; performance on variable scales or much larger images remains to be seen.
  • GPU Utilization: While the model is computationally efficient in terms of FLOPs, the many small kernels used by Mamba can lead to idle GPU time during single-pair inference, making it better suited for batched processing.

Future work includes expanding to broader archives, testing transferability to entirely new geographic regions, and exploring more complex scan-path sets.

Summary updated

Abstract

Cross-sensor remote sensing image retrieval aims to find scenes with the same content across heterogeneous sensors, such as SAR and optical or panchromatic (PAN) and multispectral (MS) imagery, without manual labels. Existing self-supervised methods either reconstruct sensor-specific pixel detail, align each image only with its co-registered twin, or rely on large transformers with 118-173M parameters (and up to 398M). We introduce RS-JEPA, a compact joint-embedding predictive framework built on a Vision Mamba encoder shared by both sensors. Since a state-space encoder reads an image as an ordered sequence of patches, we propose scan-path views, which treat two reading orders of the same image as two views with identical pixels and predict the hidden patches in one order from the visible patches in the other.To keep this prediction challenging, we propose Adversarial View Coupling (AVC), which learns the hardest split of the visible patches, and a neighbourhood consensus loss that aligns the scene similarities of both sensors. Extensive experiments on SAR-optical (BEN-14K, CBRSIR-VS) and PAN-MS (DSRSID) benchmarks demonstrate that RS-JEPA achieves the highest F1@5 among twelve re-implemented methods (72.92, 80.35 and 94.43) with a retrieval network of only 3.88M parameters (14.57M in training, 8-16times fewer than prior JEPA retrievers). Like prior methods, RS-JEPA trains on co-registered pairs. Our results highlight reading order as a source of self-supervised views. code: https://github.com/Tanishq251/RSJEPA.
BibTeX
@misc{rachamalla2026rs,
  author = {Tanishq Rachamalla and Srishti Kaushik and Muhammad Attique Khan and Gustau Camps-Valls and Swalpa Kumar Roy},
  title = {{RS-JEPA: Remote Sensing Joint-Embedding Predictive Architecture with Scan-Path Views for Cross-Sensor Image Retrieval}},
  year = {2026},
  eprint = {2611.14496},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14496},
}
Figure