TL;DR
RegNC-Mamba3 adapts the high-efficiency Mamba-3 architecture for multi-modal image registration by introducing a non-causal, path-invariant state space formulation. By integrating learnable Log-Gabor filters for structural awareness, it achieves state-of-the-art accuracy on challenging VIS-SAR and VIS-IR datasets while maintaining linear computational complexity.
What Problem Does This Paper Address?
Image registration—the process of overlaying two or more images of the same scene taken at different times, from different viewpoints, or by different sensors—is a foundational task in computer vision. Multi-modal registration (e.g., aligning a standard optical photo with a Synthetic Aperture Radar (SAR) or Infrared (IR) image) is particularly grueling.
The "modality gap" means that the same object looks completely different across sensors: SAR images are riddled with multiplicative speckle noise, while IR images often have soft edges and low contrast. Traditional methods often fail because they rely on pixel intensities or local gradients that aren't consistent across modalities. While Transformers have improved results via global attention, their quadratic complexity ($O(N^2)$) makes high-resolution processing expensive. State Space Models (SSMs) like Mamba offer linear complexity ($O(N)$), but they are inherently "causal" (designed for 1D sequences like text), making them a poor fit for the 2D, non-directional nature of images.
Methods
The authors introduce RegNC-Mamba3, a coarse-to-fine registration network built on two primary innovations:
-
Non-Causal State Space Principles (NC-SSP): Standard Mamba models scan images in fixed directions (e.g., top-left to bottom-right). This introduces a "causal bias" where a pixel only "sees" what came before it in the scan. The authors propose NC-SSP, which uses relative-weight normalization to eliminate long-range decay and aggregates information over all possible circular scan paths. This creates a shared hidden state where every token (pixel patch) can interact with the entire image simultaneously, regardless of scanning order.
-
Shared Structural Feature Awareness (SFA): To bridge the modality gap, the model needs to focus on structure (shapes, edges) rather than intensity. The authors embed learnable Log-Gabor filters directly into the NC-SSP blocks. These filters modulate the frequency domain to extract "phase congruency" features—a representation known to be highly invariant to illumination and sensor changes. By using complex-valued sequences, the model captures rotational state transitions and structural information more effectively than real-valued alternatives.
The overall architecture follows a coarse-to-fine strategy: it first finds rough correspondences at 1/8 resolution and then refines them through window-based attention at 1/4 and 1/2 resolutions.
Datasets
The model was evaluated on two major multi-modal benchmarks:
- VIS-SAR (OSDataset): Optical and SAR image pairs from GaoFen-3 and Google Earth. This is a high-noise environment due to SAR speckle.
- VIS-IR (DroneVehicle): Aligned Infrared and Visible light images of street scenes captured by drones in varying light conditions.
Key Insights and Ideas
- Beyond Euler Discretization: While most SSMs use simple Euler rules for discretization, this paper leverages the State Space Principles (SSP) from Mamba-3. This uses a more accurate trapezoidal approximation, significantly reducing truncation errors over long sequences (like large images).
- Path Invariance is Key for Registration: In registration, the geometric relationship between images is unknown. If a model's feature extraction depends on a specific scan direction, a rotated image might produce different features. By making the SSP "non-causal" and path-invariant, the authors ensure the global receptive field is spatially consistent.
- Frequency Domain Integration: Instead of just stacking more convolutional layers, the authors go back to classical signal processing. Integrating Log-Gabor filters as learnable components allows the network to "learn" the best frequency-domain modulation for cross-modal alignment.
Key Results
RegNC-Mamba3 significantly outperformed both detector-based (e.g., RIFT) and semi-dense (e.g., LoFTR, XoFTR) methods:
- Accuracy: On the VIS-SAR dataset, it achieved a Successful Matching Rate (SMR) of 97.6%, dwarfing the next best method (ReDFeat at 75.7%).
- Precision: It achieved a root mean square error (aRMSE) of 1.901 pixels on VIS-SAR and 0.911 pixels on VIS-IR, the lowest among all tested models.
- Efficiency: Compared to a "vanilla" multi-directional Mamba-3 baseline, the NC-SSP formulation is 7x faster and uses roughly 44% less GPU memory because it avoids redundant small-kernel operations and multiple scan-path computations.
Limitations and Future Work
While the model is much faster than standard Mamba-3 implementations, the authors note that it still incurs higher latency than extremely lightweight models like JamMa. This is attributed to kernel-launch overheads and memory-bandwidth limitations inherent in the complex matrix operations of NC-SSP.
The authors suggest that future work should focus on reducing this computational overhead—perhaps through more optimized CUDA kernels—to maintain high registration gains while achieving real-time performance on edge devices. Additionally, extending this non-causal principle to other 3D or volumetric multi-modal tasks (like medical imaging) remains an open opportunity.
Summary updated