Abstract
State-of-the-art deep ISP models alleviate the dilemma of limited generalization capabilities across heterogeneous inputs by increasing the size and complexity of the network, which inevitably leads to considerable growth in parameter counts and FLOPs. To address this challenge, this paper presents MetaISP - a streamlined model that achieves superior reconstruction quality by adaptively modulating its parameters and architecture in response to diverse inputs. Our rationale revolves around obtaining corresponding spatial and channel-wise correction matrices for various inputs within distinct feature spaces, which assists in assigning optimal attention. This is achieved by predicting dynamic weights for each input image and combining these weights with multiple learnable basis matrices to construct the correction matrices. The proposed MetaISP makes it possible to obtain best performance while being computationally efficient. SOTA results are achieved on two large-scale datasets, e.g. 23.80dB PSNR on ZRR, exceeding the previous SOTA 0.19dB with only 9.2% of its parameter count and 10.6% of its FLOPs; 25.06dB PSNR on MAI21, exceeding the previous SOTA 0.17dB with only 0.9% of its parameter count and 2.7% of its FLOPs.
Introduction
Traditional Image Signal Processing (ISP) pipelines—the software that turns "raw" sensor data into the photos we see—are often a complex chain of manually designed modules. While deep learning has recently replaced these with end-to-end models, researchers face a frustrating trade-off: heavy models achieve great quality but are too slow and memory-intensive for mobile devices, while lightweight models fit on phones but struggle to generalize across diverse lighting conditions and sensor types.
The authors of MetaISP aim to break this trade-off. They argue that instead of making a network bigger to handle different environments, we should make it smarter by allowing it to adapt its own parameters on the fly for every image it processes.
Methods
MetaISP uses a U-Net-style architecture, but replaces standard static layers with four novel "Meta" blocks. The core innovation is the use of basis matrices and vectors: the model learns a set of fundamental transformations, and for each input image, a small "Predict Module" decides how to mix these bases to create a custom correction for that specific photo.
- Meta Channel Correction Block (MCCB): Instead of standard channel attention, it predicts weights to combine multiple learnable basis matrices. This allows for complex inter-channel relationships with very low overhead.
- Meta Spatial Correction Block (MSCB): This provides global context (similar to self-attention in Transformers) but with linear complexity $O(HW)$. It calculates the cosine similarity between pixels and a dynamically generated "correction vector" to highlight relevant spatial regions.
- Meta Feature Extraction Block (MFEB): This block adaptively weights five predefined kernels (like Sobel for edges or Gaussian for blur) to extract the most relevant features from the RAW input.
- Meta Reconstruction Block (MRB): It uses information from the deepest layer of the U-Net to guide the final image generation, ensuring global color and tone consistency.
Experimental Setup
The model was evaluated on two major benchmarks:
- ZRR (Zurich RAW to RGB): A large-scale dataset of 48,000 RAW-sRGB pairs (448x448 resolution). It is typically used to test heavy, high-performance models.
- MAI21: A dataset of 24,000 pairs (256x256 resolution) specifically designed for the Mobile AI challenge, focusing on efficiency for on-device deployment.
Key Results and Discussion
MetaISP achieved "State-of-the-Art" (SOTA) results while being significantly lighter than competitors across both benchmarks.
- On ZRR: It reached 23.80dB PSNR, beating the previous best model by 0.19dB while using only 9.2% of the parameters and 10.6% of the FLOPs.
- On MAI21: It reached 25.06dB PSNR, exceeding the previous SOTA by 0.17dB using a staggering 0.9% of the parameters and 2.7% of the FLOPs.
- Speed: In real-world testing, MetaISP processed 1120x1120 images at 7 FPS, more than double the speed of the previous leading lightweight model (LiteISP), which managed only 3 FPS.
The results prove that adaptivity is superior to scale. A small model that dynamically adjusts its weights can outperform static models that are 100x larger. Furthermore, by using cosine similarity for spatial attention (MSCB), the authors achieved long-range modeling benefits without the quadratic computational cost typical of Transformers.
Conclusion
MetaISP demonstrates that efficient RAW-to-sRGB mapping is possible with minimal parameters by leveraging dynamic modulation. While the model shows impressive efficiency, the authors note that future work could involve further optimizing these "meta" operations for specific mobile NPU (Neural Processing Unit) architectures to maximize on-device performance. Additionally, they found that increasing the number of basis matrices beyond 10 yields diminishing returns, suggesting an upper limit to the benefits of this specific form of adaptivity.
Summary updated