Overparametrization of HyperNetworks at Fixed FLOP-Count Enables Fast Neural Image Enhancement

Lorenz K. Müller

Abstract

Deep convolutional neural networks excel at image enhancement tasks like demosaicing, denoising, and super-resolution. However, their deployment on mobile devices is often hindered by high computational costs (FLOPs). In standard convolutions, reducing FLOPs inherently reduces parameter count, which conflicts with the finding that over-parameterized networks generalize better. We propose using HyperNetworks to break this fixed ratio. This approach allows us to exceed state-of-the-art performance on the Zurich RAW-to-DSLR (ZRR) dataset with over 10x fewer FLOPs. We observe generalization curves consistent with "double-descent" behavior at a fixed FLOP-count. Furthermore, we demonstrate that this technique can reduce the computational cost of existing networks like VDN while maintaining fidelity on the SIDD dataset.

The FLOP Tax in Mobile Image Enhancement

Smartphone photography relies heavily on Deep Convolutional Neural Networks (ConvNets) to bridge the gap between small mobile sensors and high-end DSLRs. However, these models face a fundamental hardware constraint: the "FLOP Tax." In a standard convolution layer, the number of parameters and the number of operations (FLOPs) are inextricably linked. Increasing the model's capacity to "understand" complex data typically requires more channels or layers, causing FLOP counts to skyrocket beyond the thermal and battery limits of mobile devices.

Recent deep learning theory suggests that "overparametrized" models—those with significantly more parameters than training samples—often generalize better, a phenomenon known as "double descent." The challenge for mobile AI is to leverage this overparametrization without increasing the computational burden.

Methodology: Decoupling Parameters from Compute

To address this, we introduce the HyperConvolution (HC) block. Unlike standard convolutions with fixed learned weights, the HC block uses a small sub-network—a Multi-Layer Perceptron (MLP)—to dynamically predict convolutional filters for each input image.

The HyperConvolution block (left) uses an MLP to predict filters based on global context. The overall architecture (right) allows for massive parameter scaling without increasing FLOPs.

The process works as follows:

  1. Global Context: The input is compressed into a single vector via global max pooling, capturing the overall "mood" or lighting of the image.
  2. HyperNetwork Prediction: This vector passes through an MLP to generate the weights for the subsequent convolution layer.
  3. Decoupling: Because the MLP operates on a single pooled vector rather than the full-resolution image, its computational cost is independent of the image resolution. We can add millions of parameters to this MLP, significantly increasing model capacity, while the FLOP count remains nearly constant.
  4. Grouped Convolutions: These predicted filters are applied efficiently using grouped convolutions, treating each image in a batch as its own group.

Empirical Results and the Double Descent Phenomenon

We evaluated our approach on the Zurich RAW-to-DSLR (ZRR) and Smartphone Image Denoising (SIDD) datasets. A key finding was the observation of "double descent" behavior while keeping the total FLOP count perfectly flat.

Generalization error (1-MS-SSIM) vs. parameter count at a fixed FLOP-count. Note the "double descent" bump and the subsequent improvement as parameters increase, while GFLOPs (red line) remain constant.

On the ZRR task, our architecture outperformed previous state-of-the-art models like PyNet and AWNet in MS-SSIM metrics while requiring over 10x fewer FLOPs. Specifically, one version of our model achieved competitive results using only 0.1 TFLOPs, compared to the 43 TFLOPs required by PyNet.

When applied to the Variational Denoising Network (VDN) for the SIDD dataset, HyperConvolutions reduced the FLOP count by 6.8x and lowered memory usage without sacrificing denoising fidelity.

Discussion and Limitations

While HyperNetworks offer a "clever shortcut" to high-performance mobile AI, there are trade-offs. Increasing parameters significantly grows the model's storage footprint (e.g., .pth or .onnx files), which may impact app download sizes. Additionally, the efficiency of grouped convolutions can vary across different hardware backends. Finally, we observed that the "forward" path of the network must maintain sufficient width; if the main convolutional backbone is too narrow, an "information bottleneck" occurs that no amount of HyperNetwork overparametrization can overcome.

In conclusion, breaking the rigid link between parameters and FLOPs via HyperNetworks provides a powerful tool for deploying state-of-the-art image enhancement on compute-constrained mobile hardware.

Summary updated

Abstract

Deep convolutional neural networks can enhance images taken with small mobile camera sensors and excel at tasks like demoisaicing, denoising and super-resolution. However, for practical use on mobile devices these networks often require too many FLOPs and reducing the FLOPs of a convolution layer, also reduces its parameter count. This is problematic in view of the recent finding that heavily over-parameterized neural networks are often the ones that generalize best. In this paper we propose to use HyperNetworks to break the fixed ratio of FLOPs to parameters of standard convolutions. This allows us to exceed previous state-of-the-art architectures in SSIM and MS-SSIM on the Zurich RAW-to-DSLR (ZRR) data-set at > 10x reduced FLOP-count. On ZRR we further observe generalization curves consistent with 'double-descent' behavior at fixed FLOP-count, in the large image limit. Finally we demonstrate the same technique can be applied to an existing network (VDN) to reduce its computational cost while maintaining fidelity on the Smartphone Image Denoising Dataset (SIDD). Code for key functions is given in the supplemental.
BibTeX
@inproceedings{muller2021overparametrization,
  author = {Lorenz K. Müller},
  title = {{Overparametrization of HyperNetworks at Fixed FLOP-Count Enables Fast Neural Image Enhancement}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=x9ziQtKtlRM},
}
Figure