TL;DR
QMambaBSR introduces a novel framework for burst image super-resolution by adapting State Space Models (Mamba) for multi-frame information retrieval. By utilizing a Query State Space Model (QSSM) to jointly query inter-frame data and scan intra-frame features, the network effectively separates consistent sub-pixel details from random noise. Coupled with an Adaptive Upsampling (AdaUp) module, it achieves state-of-the-art performance, reconstructing sharp, artifact-free high-resolution images from low-resolution bursts.
What Problem Does This Paper Address?
When you take a quick succession of photos on a smartphone (a burst), slight hand tremors cause minor sub-pixel shifts between frames. Burst Image Super-Resolution (BurstSR) exploits these shifts to reconstruct a single, highly detailed, high-resolution (HR) image.
However, two major challenges plague existing BurstSR methods:
- Noise vs. Detail Dilemma: Burst images contain high-frequency random noise. Traditional methods fuse frames either by simple concatenation or frame-by-frame processing. This makes it incredibly difficult to distinguish actual sub-pixel details from random noise outliers, often leading to artifacts.
- Static Upsampling Blurring: Most architectures rely on static upsampling techniques (like pixel shuffle or fixed interpolations) during the final reconstruction phase. Because these methods apply the same operations regardless of the scene, they fail to adapt to varying sub-pixel distributions, resulting in over-smoothed textures.
Methods
QMambaBSR addresses these bottlenecks through a specialized architecture consisting of three core components:
1. Query State Space Model (QSSM)
Standard State Space Models (SSMs) like Mamba excel at processing sequences but lack multi-frame querying mechanisms. The authors modify the discretization process of the SSM so that the base frame (the anchor image) dynamically controls the matrix $B$ and the step size $\Delta$. Meanwhile, the current frames (the remaining burst images) drive the state transitions. This allows the base frame to act as a gate, querying all current frames simultaneously to extract clean sub-pixel data while suppressing random noise in a single step.
2. Multi-Scale Fusion Module (MSFM)
To handle details at various scales, MSFM runs three parallel branches:
- A CNN branch for local feature extraction.
- An SSM branch with horizontal and vertical scanning for structural patterns.
- A Channel Transformer branch to capture long-range global context.
3. Adaptive Upsampling Module (AdaUp)
Instead of using fixed upsampling filters, AdaUp uses adaptive pooling and sequence feature interaction to perceive the spatial distribution of sub-pixels in the feature space. It dynamically generates a scene-specific upsampling kernel, ensuring that spatial arrangements are accurately reconstructed without over-smoothing.
Datasets
The authors evaluated their method on four widely accepted BurstSR benchmarks:
- Synthetic BurstSR: A large-scale synthetic dataset used for baseline training and evaluation.
- Real BurstSR: Real-world burst images used to test generalization.
- RealBSR-RAW: A real-world dataset consisting of raw camera sensor data.
- RealBSR-RGB: A real-world dataset processed into standard RGB color space.
Key Insights and Ideas
- Exploiting Noise Randomness: The foundational insight of this paper is that while true sub-pixel details maintain a consistent spatial distribution and intensity across burst frames, high-frequency noise appears completely at random (as illustrated in Figure 1). By processing the entire sequence globally rather than frame-by-frame, the network easily flags and eliminates noise outliers.
- Mamba as a Query Engine: Instead of relying on computationally expensive cross-attention mechanisms to align and query features, QMambaBSR elegantly maps the query-key-value paradigm onto the parameters of a linear State Space Model, drastically improving efficiency while maintaining a progressively diminishing receptive field that favors localized neighborhood interactions.
Key Results
QMambaBSR achieves a new state-of-the-art across both synthetic and real-world datasets:
- Synthetic BurstSR: Outperforms the previous best method (Burstormer) by 0.29 dB in PSNR, reaching an impressive 43.12 dB.
- RealBSR-RAW: Surpasses Burstormer by 0.268 dB in PSNR.
- RealBSR-RGB: Surpasses Burstormer by 0.204 dB in PSNR.
Ablation Study Highlights
An ablation study isolating the core modules demonstrated their individual impact on a baseline network:
- Adding the MSFM module improved performance by 1.34 dB.
- Integrating the QSSM added another 0.72 dB.
- Incorporating the AdaUp module contributed an additional 0.26 dB, validating the choice of dynamic upsampling over static alternatives.
Limitations and Future Work
While QMambaBSR excels at the fusion and upsampling stages of the pipeline, it still relies on external, standard alignment blocks to pre-align the burst frames. The authors note that their future work will focus on:
- Utilizing State Space Models to inherently enhance and integrate the alignment stage directly into the burst processing pipeline.
- Exploring the architecture's utility in other burst-frame restoration tasks, such as burst denoising and High Dynamic Range (HDR) imaging.
Summary updated