TL;DR
This paper introduces an "Expanded Convolutional Network" (ECN) that uses a novel "Expanded Convolution" (EC) layer to efficiently enlarge the receptive field (RF) of super-resolution (SR) models. By merging multiple look-up tables (LUTs) into a single one and optimizing the output scaling, their ECLUT method achieves significantly faster inference speeds than existing LUT-based SR methods while maintaining comparable image quality, making it ideal for edge devices.
What Problem Does This Paper Address?
Single Image Super-Resolution (SISR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs, with applications ranging from medical imaging to surveillance. While deep learning methods have achieved impressive results, they often demand substantial computational resources and memory, limiting their deployment on resource-constrained edge devices. Look-up tables (LUTs) offer a promising alternative by trading computation for memory, enabling faster inference. However, traditional LUT-based SR methods face a trade-off: increasing the receptive field (RF) to improve image quality exponentially increases LUT size, leading to memory issues and slower retrieval. Existing methods that try to mitigate this by using multiple LUTs or complex indexing schemes often sacrifice inference speed. This paper addresses the challenge of achieving high-quality SR with significantly reduced computational and memory footprints, specifically targeting real-time performance on edge devices.
Methods
The core innovation is the Expanded Convolution (EC) layer. Unlike traditional convolutions that map an input patch to a single output pixel, the EC layer expands the output size of a convolution operation. This is achieved by rearranging and padding the output feature maps, effectively creating an overlapping sliding window at the output end. The overlapping parts are then summed to produce the final output. This process indirectly enlarges the receptive field (RF) of the network.
The paper builds upon the SR-LUT approach, which trains an SR network and then stores its input-output mapping in a LUT for fast inference. The key modifications are:
- Expanded Convolution (EC) Layer: Placed at the end of the network, this layer expands the output, allowing a single LUT to cover a larger RF. This is crucial because it makes the LUT size grow linearly with the RF increase, unlike the exponential growth seen in traditional methods.
- Single LUT Merging: The EC layer's output, which covers the target pixel and its surrounding regions, allows for the merging of multiple potential LUT entries into a single one. This significantly reduces the number of LUTs needed, leading to faster lookups.
- Rescale Output: To mitigate quantization errors introduced when storing intermediate results in an 8-bit LUT, the authors propose expanding the numerical range of the accumulated results and then rescaling them back to the standard [0, 255] range. This is achieved using a simple scaling factor and rounding, which has negligible computational cost.
Datasets
The paper uses the following datasets:
- Training: DIV2K dataset (800 images for training, 100 for validation).
- Evaluation:
- Set5
- Set14
- BSDS100
- Urban100
- Mang109
No new benchmarks were introduced.
Key Insights and Ideas
- Receptive Field Expansion at the Output: The paper cleverly shifts the RF expansion from the input side to the output side using the EC layer. This is a critical insight because it allows for a linear increase in LUT size with RF, overcoming a major bottleneck in traditional LUT-based SR.
- Single LUT for Speed: By enabling the merging of multiple potential lookups into one through the EC layer, the method drastically reduces the number of LUTs required. This directly translates to faster inference, as memory access is a primary bottleneck in LUT-based methods.
- Quantization Error Mitigation: The simple yet effective rescaling of intermediate results before final quantization is a practical solution to a common problem in LUT-based systems. It shows that even small adjustments can significantly recover performance lost due to precision reduction.
- Hardware-Friendly Approach: The entire inference process avoids floating-point operations, relying solely on LUT lookups and simple arithmetic (addition and bit shifts for rescaling). This makes the method highly suitable for deployment on low-power edge devices.
Key Results
- Inference Speed: The proposed ECLUT method achieves significantly faster inference times compared to other LUT-based SR methods (e.g., SR-LUT, SP-LUT, MuLUT). It is only slightly slower than basic interpolation methods like nearest neighbor and bilinear. For example, it achieves a runtime of 41ms for generating a 1280x720 image, while other LUT methods range from 94ms to over 4955ms.
- Image Quality: The method achieves comparable PSNR and SSIM scores to other advanced LUT-based methods. For instance, compared to bicubic interpolation, ECLUT shows improvements of +1.49dB PSNR and +0.036 SSIM on the Set5 dataset. While deep learning methods like RRDB achieve higher PSNR, their inference times are orders of magnitude slower.
- Storage Efficiency: The single LUT approach, enabled by the EC layer, leads to a manageable storage size (9MB for the proposed method vs. 1.274MB for SR-LUT, 5.5MB for SP-LUT, and 4.062MB for MuLUT, though the paper notes SR-LUT's size is for a different model configuration).
Limitations and Future Work
- Indexing Capability: The paper acknowledges that their current implementation uses only two pixels for single indexing, which limits its indexing capability compared to methods like MuLUT. This is a deliberate choice to prioritize speed.
- Future Work: The authors suggest exploring the application of the EC method in conjunction with multiple LUTs to further improve SR performance, potentially balancing speed and quality even further. They also mention that increasing the number of pixels used for single indexing can gradually improve performance, as shown in their ablation studies.
Summary updated