Abstract
Deep convolutional neural networks excel at image enhancement tasks like demosaicing, denoising, and super-resolution. However, their deployment on mobile devices is often hindered by high computational costs (FLOPs). In standard convolutions, reducing FLOPs inherently reduces parameter count, which conflicts with the finding that over-parameterized networks generalize better. We propose using HyperNetworks to break this fixed ratio. This approach allows us to exceed state-of-the-art performance on the Zurich RAW-to-DSLR (ZRR) dataset with over 10x fewer FLOPs. We observe generalization curves consistent with "double-descent" behavior at a fixed FLOP-count. Furthermore, we demonstrate that this technique can reduce the computational cost of existing networks like VDN while maintaining fidelity on the SIDD dataset.
The FLOP Tax in Mobile Image Enhancement
Smartphone photography relies heavily on Deep Convolutional Neural Networks (ConvNets) to bridge the gap between small mobile sensors and high-end DSLRs. However, these models face a fundamental hardware constraint: the "FLOP Tax." In a standard convolution layer, the number of parameters and the number of operations (FLOPs) are inextricably linked. Increasing the model's capacity to "understand" complex data typically requires more channels or layers, causing FLOP counts to skyrocket beyond the thermal and battery limits of mobile devices.
Recent deep learning theory suggests that "overparametrized" models—those with significantly more parameters than training samples—often generalize better, a phenomenon known as "double descent." The challenge for mobile AI is to leverage this overparametrization without increasing the computational burden.
Methodology: Decoupling Parameters from Compute
To address this, we introduce the HyperConvolution (HC) block. Unlike standard convolutions with fixed learned weights, the HC block uses a small sub-network—a Multi-Layer Perceptron (MLP)—to dynamically predict convolutional filters for each input image.
The process works as follows:
- Global Context: The input is compressed into a single vector via global max pooling, capturing the overall "mood" or lighting of the image.
- HyperNetwork Prediction: This vector passes through an MLP to generate the weights for the subsequent convolution layer.
- Decoupling: Because the MLP operates on a single pooled vector rather than the full-resolution image, its computational cost is independent of the image resolution. We can add millions of parameters to this MLP, significantly increasing model capacity, while the FLOP count remains nearly constant.
- Grouped Convolutions: These predicted filters are applied efficiently using grouped convolutions, treating each image in a batch as its own group.
Empirical Results and the Double Descent Phenomenon
We evaluated our approach on the Zurich RAW-to-DSLR (ZRR) and Smartphone Image Denoising (SIDD) datasets. A key finding was the observation of "double descent" behavior while keeping the total FLOP count perfectly flat.
On the ZRR task, our architecture outperformed previous state-of-the-art models like PyNet and AWNet in MS-SSIM metrics while requiring over 10x fewer FLOPs. Specifically, one version of our model achieved competitive results using only 0.1 TFLOPs, compared to the 43 TFLOPs required by PyNet.
When applied to the Variational Denoising Network (VDN) for the SIDD dataset, HyperConvolutions reduced the FLOP count by 6.8x and lowered memory usage without sacrificing denoising fidelity.
Discussion and Limitations
While HyperNetworks offer a "clever shortcut" to high-performance mobile AI, there are trade-offs. Increasing parameters significantly grows the model's storage footprint (e.g., .pth or .onnx files), which may impact app download sizes. Additionally, the efficiency of grouped convolutions can vary across different hardware backends. Finally, we observed that the "forward" path of the network must maintain sufficient width; if the main convolutional backbone is too narrow, an "information bottleneck" occurs that no amount of HyperNetwork overparametrization can overcome.
In conclusion, breaking the rigid link between parameters and FLOPs via HyperNetworks provides a powerful tool for deploying state-of-the-art image enhancement on compute-constrained mobile hardware.
Summary updated