Abstract
Implicit neural representations have recently been extended to represent convolutional neural network weights via neural representation for neural networks (NeRN), offering promising parameter compression benefits. However, standard multi-layer perceptrons (MLPs) used in NeRN exhibit a pronounced spectral bias, hampering their ability to reconstruct high-frequency details effectively. In this paper, we propose SBS, a parameter-efficient enhancement to NeRN that suppresses spectral bias using two techniques: (1) a unidirectional ordering-based smoothing (UOS) that improves kernel smoothness in the output space, and (2) UOS-aware random fourier features (UOS-RFF) that adaptively modulate the frequency bandwidth of input encodings based on layer-wise parameter count. Extensive evaluations on various ResNet models with datasets CIFAR-10, CIFAR-100, and ImageNet, demonstrate that SBS achieves significantly better reconstruction accuracy with fewer parameters compared to the state-of-the-art.
Introduction
Implicit Neural Representations (INRs) have revolutionized the storage of 3D scenes and images by representing them as continuous functions learned by coordinate-based MLPs. Recently, this concept was extended to Neural Representation for Neural Networks (NeRN), where a compact MLP generates the weights of a much larger Convolutional Neural Network (CNN). This paradigm offers massive potential for model compression and mobile deployment.
However, a fundamental challenge remains: Spectral Bias. Standard MLPs act as low-pass filters, naturally learning smooth, low-frequency patterns while struggling to capture the high-frequency details often present in CNN weights. Previous attempts to capture these details required larger, less efficient MLPs. SBS addresses this by suppressing spectral bias to make weight-generating MLPs both smaller and more accurate.
The SBS Framework
SBS optimizes the NeRN pipeline across both the "output space" (the weights being generated) and the "input space" (the coordinates fed into the MLP).
Unidirectional Ordering-based Smoothing (UOS)
CNN weights are inherently non-smooth, making them difficult for MLPs to fit. While previous work attempted multi-directional weight permutation, SBS argues that this causes "interference" that degrades performance. Instead, SBS employs a greedy algorithm to order weights along a single, unidirectional path based on Euclidean distance. This maximizes the low-frequency energy of the signal, creating a smoother target that the MLP can digest more efficiently.
UOS-aware Random Fourier Features (UOS-RFF)
To help the MLP capture necessary high frequencies, researchers often use Random Fourier Features (RFF) for input encoding. SBS introduces a dynamic approach: it adaptively modulates the frequency bandwidth ($\sigma^2$) based on the layer-wise parameter count. The authors discovered an inverse relationship where wider CNN layers require a lower RFF bandwidth to maintain the optimal balance between smoothness and detail.
Theoretical Motivation and Spectral Analysis
The paper utilizes Neural Tangent Kernel (NTK) theory to mathematically demonstrate why standard MLPs fail to capture high-frequency weight details. By applying UOS, the magnitude spectrum of the weights is shifted toward the low-frequency range, as shown in the spectral analysis. This alignment between the data's frequency distribution and the MLP's natural learning bias is the core of SBS's efficiency.
Experimental Results
SBS was evaluated on standard image classification benchmarks using ResNet architectures:
- CIFAR-10/100: On ResNet-20 (CIFAR-10), SBS achieved 90.50% accuracy using only 20% of the original parameters, significantly outperforming the baseline NeRN's 87.42% at the same compression level.
- ImageNet: For ResNet-18, SBS reached 68.57% accuracy at a 15% compression ratio. Notably, SBS outperformed NeRN variants that utilized stronger "teacher" networks for guidance, despite SBS not requiring such external supervision.
Overall, the method provides a 2x–3x improvement in parameter efficiency compared to previous state-of-the-art methods.
Conclusion and Future Work
SBS successfully mitigates spectral bias in NeRN through targeted smoothing and adaptive feature encoding. While the system introduces a small metadata overhead (approx. 4%) for storing permutation indices and requires intensive training (30-50 hours on a V100 for ImageNet), the gains in parameter efficiency are substantial. Future research may extend these techniques beyond ResNets to architectures like Vision Transformers (ViTs).
Summary updated