TL;DR
RS-JEPA is a compact, self-supervised framework for cross-sensor remote sensing image retrieval that leverages the unique sequence-dependent properties of Vision Mamba. By treating different image reading orders as "scan-path views" and using adversarial training to prevent shortcuts, it achieves state-of-the-art accuracy with a retrieval network of just 3.88M parameters—up to 16x smaller than previous transformer-based architectures.
What Problem Does This Paper Address?
In remote sensing, we often need to find the same scene across different sensor types—for example, matching a cloudy Synthetic Aperture Radar (SAR) image to a clear-sky optical image. This is a "cross-sensor retrieval" problem.
It is notoriously difficult for three reasons:
- Heterogeneity: The same physical scene looks radically different across sensors (e.g., SAR speckle vs. optical color).
- Label Scarcity: Manually labeling millions of satellite images is impossible, necessitating self-supervised learning.
- Efficiency: Large-scale archives require small, fast encoders. Existing methods often rely on massive Vision Transformers (ViTs) that are computationally expensive to run over entire archives.
The authors identify a gap: while Joint-Embedding Predictive Architectures (JEPAs) have shown promise by predicting features instead of pixels (avoiding the trap of learning sensor-specific noise), they have remained bulky.
Methods
The core innovation of RS-JEPA lies in its use of Vision Mamba (Vim), a state-space model (SSM). Unlike Transformers, which are order-agnostic due to global attention, SSMs process images as ordered sequences. This "architectural bias" becomes a feature, not a bug.
1. Scan-Path Views
Because a Mamba encoder's output for a specific patch depends on the tokens it has already "read," the authors propose using different traversal orders (e.g., row-major vs. column-major) of the same image as distinct views. This provides a natural source of self-supervision without needing complex image augmentations or momentum encoders.
2. Adversarial View Coupling (AVC)
To prevent the model from simply interpolating nearby pixels (a "trivial" task), the authors introduced Adversarial View Coupling. A scorer learns to split the visible patches into two halves that are the hardest to use for predicting hidden patches. This forces the model to learn global scene context rather than local spatial continuity.
3. Neighborhood Consensus Loss
Beyond aligning a SAR image with its exact optical twin, RS-JEPA uses a consensus loss to ensure that the "neighborhood geometry" is preserved. If two scenes are similar in the SAR domain, their embeddings should also be similar in the optical domain.
Datasets
The model was evaluated on three major dual-sensor benchmarks:
- BEN-14K: A subset of BigEarthNet featuring Sentinel-1 (SAR) and Sentinel-2 (Optical) pairs.
- CBRSIR-VS: SAR and RGB image pairs.
- DSRSID: High-resolution panchromatic (PAN) and lower-resolution multispectral (MS) images.
Key Insights and Ideas
- Reading Order as Supervision: The most elegant "aha moment" is realizing that for sequence-based models like Mamba, the path taken through the pixels is a valid self-supervised signal.
- Feature Prediction > Pixel Reconstruction: By predicting latent features (JEPA style) rather than raw pixels (MAE style), the model ignores sensor-specific "noise" like SAR speckle, focusing instead on the underlying semantic content.
- Small is Beautiful: The paper demonstrates that a carefully designed 3.88M parameter model can outperform 100M+ parameter models by better leveraging architectural priors.
Key Results
RS-JEPA achieved the highest F1@5 scores across all three benchmarks, outperforming twelve other re-implemented state-of-the-art methods.
- Accuracy: It reached 72.92 (BEN-14K), 80.35 (CBRSIR-VS), and 94.43 (DSRSID) in mean F1@5.
- Efficiency: The retrieval network uses only 3.88M parameters, which is 8–16× fewer than prior JEPA-based retrievers.
- Throughput: As shown in Figure 1, RS-JEPA sits in the "sweet spot" of the Pareto front—offering the best retrieval accuracy while maintaining high throughput.
Limitations and Future Work
While impressive, the authors note a few limitations:
- Co-registration Requirement: The model still requires co-registered image pairs for training, which can be a bottleneck for scaling to unaligned datasets.
- Fixed Resolution: Experiments were conducted at a fixed 120x120 resolution; performance on variable scales or much larger images remains to be seen.
- GPU Utilization: While the model is computationally efficient in terms of FLOPs, the many small kernels used by Mamba can lead to idle GPU time during single-pair inference, making it better suited for batched processing.
Future work includes expanding to broader archives, testing transferability to entirely new geographic regions, and exploring more complex scan-path sets.
Summary updated