TL;DR
This paper bridges the gap between two popular methods for representing high-frequency signals with neural networks: SIRENs and Fourier feature mappings. They demonstrate that a Fourier-mapped perceptron is structurally equivalent to a single-layer SIREN, and introduce an "integer lattice mapping" that forces network outputs to be periodic, akin to a Fourier series. This new mapping, combined with a progressive training strategy, improves generalization and outperforms existing methods in image regression and novel view synthesis tasks.
What Problem Does This Paper Address?
Implicit Neural Representations (INRs) have shown great promise in representing complex, high-resolution signals like 3D shapes and images using neural networks. However, traditional neural networks with ReLU activations struggle to capture high-frequency details. Two prominent solutions have emerged: SIRENs, which use sinusoidal activation functions, and Fourier feature mappings, which transform input coordinates using sinusoidal functions before feeding them into a standard network. While both are effective, their underlying connections and optimal usage, especially regarding overfitting and generalization, remained unclear. This paper aims to demystify these connections and propose a more principled approach to Fourier-based INR.
Methods
The core of this paper lies in establishing a mathematical equivalence between Fourier mappings and SIRENs. They show that a perceptron with a specific Fourier mapping applied to its input is structurally identical to a single-layer SIREN.
Their key contribution is the integer lattice mapping. This mapping is derived from the d-dimensional Fourier series expansion. By restricting the mapping parameters to integers, they effectively force the network's output to be periodic. This is a significant departure from random Fourier mappings, which can introduce arbitrary frequencies.
They also adapt a progressive training strategy (originally proposed for NeRFs) to their integer lattice mapping. This strategy involves gradually introducing higher frequencies during training, starting with low frequencies. This helps prevent overfitting and improves generalization.
Finally, they explore pruning of the integer lattice mapping, selecting only the most relevant frequencies based on learned weights, and investigate the application of these mappings within Multi-Layer Perceptrons (MLPs).
Datasets
The paper uses standard image datasets for their experiments. Specifically, they mention using "ten images with a resolution of 512 × 512" for image regression. For novel view synthesis, they use the "bulldozer scene," a common benchmark for NeRF-like experiments.
Key Insights and Ideas
- SIRENs and Fourier Mappings are Fundamentally Linked: The paper provides a clear mathematical link, showing that a Fourier-mapped perceptron is essentially a SIREN with a fixed, non-trainable mapping. This explains why both approaches are effective for high-frequency signals.
- Integer Lattice Mapping Enforces Periodicity: By constraining the Fourier mapping to integer frequencies, the network's output becomes inherently periodic. This is a powerful insight, as many real-world signals can be viewed as periodic over their bounded domains.
- Progressive Training Mitigates Overfitting: Gradually introducing frequencies during training, rather than all at once, significantly improves generalization and prevents the network from simply memorizing the training data.
- Pruning Can Improve Performance: Selecting the most impactful frequencies through pruning can lead to better results, especially when dealing with limited memory or computational resources.
- Mapping Choice Matters for MLPs: While a good mapping might work well for a single perceptron, its effectiveness can change when incorporated into deeper MLPs, suggesting that the interaction between mapping and network architecture is complex.
Key Results
- Perceptron as Fourier Series: The integer lattice mapping, when applied to a perceptron, is shown to be equivalent to a Fourier series.
- Improved Generalization with Progressive Training: Experiments show that progressive training consistently leads to higher test PSNR scores in image regression tasks, indicating better generalization.
- Integer Mapping Outperforms Others: For higher mapping frequencies (N), the integer lattice mapping consistently outperforms SIREN, Gaussian, and positional encoding methods in image regression, achieving higher PSNR.
- Qualitative Improvements in View Synthesis: In novel view synthesis, pruning the integer mapping leads to qualitatively better results, reducing artifacts and producing more realistic renderings.
- Mapping Size and Standard Deviation are Key: Consistent with prior work, the paper confirms that the size of the mapping (number of frequencies) and the standard deviation of its elements are the primary drivers of performance.
Limitations and Future Work
- Memory Constraints: The paper acknowledges that memory limitations prevented them from testing very high frequencies in the novel view synthesis experiments, which were shown to be superior in image regression.
- Learning Schedules: The authors mention that they did not extensively study different learning schedules for progressive training and suggest this as a direction for future work.
- MLP Performance Nuances: The paper notes that the best mapping for a perceptron doesn't always translate to the best mapping for a deeper MLP, indicating a need for further research into how mappings interact with network depth and width.
- Exploration of Other Activation Functions: While they compare sine and ReLU activations, further exploration with other activation functions in conjunction with their integer mapping could be beneficial.
Summary updated