Abstract
Look-Up Table (LUT) has recently gained increasing attention for restoring High-Quality (HQ) images from Low-Quality (LQ) observations, thanks to its high computational efficiency achieved through a “space for time” strategy of caching learned LQ-HQ pairs. However, incorporating multiple LUTs for improved performance comes at the cost of a rapidly growing storage size, which is ultimately restricted by the allocatable on-device cache size. In this work, we propose a novel LUT compression framework to achieve a better trade-off between storage size and performance for LUT-based image restoration models. Based on the observation that most cached LQ image patches are distributed along the diagonal of a LUT, we devise a Diagonal-First Compression (DFC) framework, where diagonal LQ-HQ pairs are preserved and carefully re-indexed to maintain the representation capacity, while non-diagonal pairs are aggressively subsampled to save storage. Extensive experiments on representative image restoration tasks demonstrate that our DFC framework significantly reduces the storage size of LUT-based models (including our new design) while maintaining their performance. For instance, DFC saves up to 90% of storage at a negligible performance drop for x4 super-resolution. The source code is available on GitHub: https://github.com/leenas233/DFC.
Introduction
Deep Neural Networks (DNNs) are the state-of-the-art for image restoration tasks such as super-resolution, denoising, and deblurring. However, their high computational cost often makes them impractical for real-time applications on edge devices like smartphones. To address this, researchers have adopted a "space for time" strategy using Look-Up Tables (LUTs). By pre-calculating and storing DNN outputs for all possible local input patches, inference is reduced to simple memory lookups, replacing heavy floating-point operations (FLOPs).
Despite their efficiency, LUT-based methods face a significant storage bottleneck. Improving restoration quality typically requires larger receptive fields, which necessitates higher-dimensional or cascaded LUTs. This leads to a storage explosion. For example, while basic models might require ~1MB, advanced architectures can exceed 17MB. This is problematic because modern mobile System-on-Chips (SoCs), such as the Snapdragon 888, typically have L3 cache sizes limited to 2MB–4MB. When a LUT exceeds the cache size, the system must access slow main memory (DRAM), leading to a drastic drop in inference speed.
Proposed Methodology
Spatial Progressive Fusion LUT (SPF-LUT)
To break the performance ceiling of previous LUT methods, we propose SPF-LUT. Inspired by multi-scale neural networks, SPF-LUT cascades multiple LUT groups. It employs a channel-splitting strategy to progressively expand the receptive field while fusing multi-scale features. This allows the model to capture more complex spatial information than single-channel, single-layer LUTs.
Diagonal-First Compression (DFC)
The core contribution of this work is the Diagonal-First Compression (DFC) framework. It is based on the "diagonal-dominance" property of natural images: adjacent pixels in a local patch are usually highly correlated and similar in value. Consequently, the most frequently accessed cells in a LUT lie along its diagonal.
DFC exploits this redundancy through two strategies:
- Diagonal Re-indexing: Cells satisfying the diagonal condition ($|I_1 - I_0| \leq \lambda$) are preserved. An "Index Mapper" is used to store these high-priority cells in a compact, lower-dimensional format, maintaining the model's representation capacity where it matters most.
- Non-diagonal Subsampling: For off-diagonal cells representing rare pixel combinations, the framework applies aggressive subsampling at a large interval ($\sigma$) to drastically reduce storage requirements.
Experimental Evaluation
We evaluated our framework on several standard benchmarks, including DIV2K for super-resolution and GoPro for deblurring. The results demonstrate that DFC achieves massive compression with minimal impact on visual quality.
As shown in the figure below, DFC reduced the storage of the SPF-LUT model from 17.28MB to just 2.02MB for $\times4$ super-resolution—an 88.3% reduction. Crucially, this brings the model size within the 4MB L3 cache limit of modern mobile hardware. The performance drop was negligible, with only a 0.06 dB decrease in PSNR on the Set5 dataset. Furthermore, the compressed SPF-LUT+DFC outperforms uncompressed state-of-the-art models like MuLUT while requiring significantly less storage.
Conclusion
The Diagonal-First Compression framework effectively addresses the storage bottleneck of LUT-based image restoration. By prioritizing the preservation of diagonal cells that represent the most common natural image patterns, DFC enables high-performance models to operate within the strict cache constraints of mobile devices. Future work will explore automated methods for optimizing compression parameters like the diagonal width and subsampling intervals to further refine the efficiency of LUT-based restoration.
Summary updated