Beyond Heuristics: Globally Optimal Configuration of Implicit Neural Representations

Sipeng Chen, Yan Zhang, Shibo LiSubmission to ICLR 2026 · Desk rejected

TL;DR

Implicit Neural Representations (INRs) are notoriously finicky: a slight tweak in activation functions or initialization can easily swing performance by 10 dB. OptiINR formalizes INR configuration as a joint global optimization problem, using mixed-variable Bayesian optimization to simultaneously select per-layer activations, initialization scales, and learning rates. Across 1D audio, 2D images, 3D shapes, NeRFs, and PDEs, OptiINR consistently discovers hybrid network configurations that significantly outperform hand-tuned baselines.

What Problem Does This Paper Address?

Coordinate-based MLPs (INRs) represent continuous signals—such as images, 3D shapes, and audio—by mapping spatial or temporal coordinates to signal values. However, they suffer from a fundamental tension between expressivity and optimization stability known as the capacity–convergence gap.

While periodic (SIREN, FINER) or wavelet (WIRE) activations enable networks to represent high-frequency details and overcome spectral bias, they are acutely sensitive to weight initialization and layer-wise scaling. Because choices made in one layer heavily alter the gradient dynamics and spectral properties of subsequent layers, tuning them in isolation or using greedy layer-by-layer methods (like MIRE) fails to guarantee global optimality. In practice, researchers rely on brittle heuristics or coarse manual tuning.

Methods

OptiINR frames INR design as a black-box, mixed-variable global optimization problem:

  • Mixed-Variable Architecture Search Space ($\mathcal{L}$): For an $L$-layer network, the configuration vector parameterizes whether to use positional encoding (PE), categorical activation functions per layer (e.g., SIREN, WIRE, Gaussian, FINER++), continuous activation hyperparameters (e.g., frequency $\omega_0$ or spatial scale $s_0$), initialization bounds, and per-layer learning rates $\alpha_l$.
  • Composite Product Kernel: To model dependencies across discrete and continuous choices, OptiINR uses a Gaussian Process (GP) surrogate with a product kernel: $$k(\Lambda, \Lambda') = k_{\text{cont}}(\Lambda_c, \Lambda'_c) \times k_{\text{cat}}(\Lambda_{\text{cat}}, \Lambda'_{\text{cat}})$$ Continuous parameters use a Matérn kernel, while one-hot encoded categorical parameters use an Automatic Relevance Determination (ARD) Squared Exponential kernel. The authors formally prove the positive semi-definiteness of this kernel via the Schur product theorem.

  • Efficient Posterior Sampling via Matheron’s Rule: OptiINR optimizes a Monte Carlo Empirical Expected Improvement (EEI) acquisition function. Instead of computing an expensive $O(S n^3)$ Cholesky factorization for $S$ samples over $n$ evaluations, it uses Matheron’s rule (conditional simulation) to invert the covariance matrix only once per iteration, amortizing the cost to $O(n^3 + S n^2)$.

  • Theoretical Grounding: The authors prove with a game-theoretic counterexample that greedy layerwise optimization is suboptimal and connect layer activation selection to the spectral properties of the network's Neural Tangent Kernel (NTK).

Datasets

The authors evaluated OptiINR across five distinct modalities and benchmarks:

  • 2D Images: The 24-image Kodak suite ($768 \times 512$) and 16 high-frequency patches from DIV2K ($512 \times 512$).
  • 1D Audio: Standard SIREN benchmarks (Bach, Counting, Two Speakers).
  • 3D Occupancy: High-resolution meshes from the Stanford 3D Scanning Repository (Dragon and Thai Statue voxelized at $512^3$).
  • Novel View Synthesis (NeRF): The synthetic Lego scene from Mildenhall et al. (2020).
  • Physics / PDEs: The Shear Flow dataset from The Well benchmark (spatiotemporal tracer field, trajectory #30).

Key Insights and Ideas

  • Search Space Over Optimizer: Surprisingly, running basic random search or grid search over OptiINR’s joint space already outperformed frequency-only tuning methods like FreSh. The biggest performance lever is allowing heterogeneous, per-layer activation choices rather than forcing a uniform activation throughout the network.
  • Smooth-to-Periodic Architectural Hierarchy: OptiINR automatically converges on intuitive, multi-scale network designs: early layers tend to adopt smooth or localized activations (Gaussian or FINER++) to capture low-frequency global structure, while deeper layers adopt periodic or wavelet activations (SIREN, WIRE) to resolve high-frequency edges and transients.
  • Layer-Dependent Learning Rates: Discovered configurations consistently prefer higher learning rates in early layers to rapidly anchor the global coordinate mapping, and smaller learning rates in deep layers for stable refinement of fine textures.

Key Results

  • Audio Reconstruction: Dramatic gains over the best baselines, reaching 60.84 dB on Bach (+5.90 dB over FR), 49.60 dB on Counting (+10.25 dB over FINER), and 68.39 dB on Two Speakers (+12.03 dB over FR).
  • 2D Image Representation: Achieves 41.38 dB average PSNR on Kodak (+1.14 dB over FINER, +2.91 dB over SIREN) and 46.24 dB on DIV2K (+0.68 dB over FINER, +3.49 dB over SIREN).
  • 3D Occupancy: Outperformed all baselines, reaching 0.9936 IoU on Dragon and 0.9884 IoU on Thai Statue (capturing fine geometric motifs and thin protrusions).
  • NeRF & PDEs: Achieved 25.63 dB on NeRF Lego (surpassing WIRE by 1.00 dB and ReLU+PE by 0.58 dB) and 57.02 dB on PDE Shear Flow (+3.06 dB over SIREN).
  • Computational Overhead: The GP surrogate and acquisition optimization add only $\sim 2.2 \times 10^6$ FLOPs per run—negligible compared to the $\sim 4.0 \times 10^{15}$ FLOPs required for full INR training, making the search easily parallelizable across GPU clusters.

Limitations and Future Work

  • High Per-Signal Compute: Because OptiINR solves an expensive black-box problem by training an INR to convergence at every proposal step (e.g., 130 trials of 10,000 epochs), finding an optimal configuration for a single image or 3D shape requires significant aggregate GPU compute.
  • Lack of Amortization / Meta-Learning: OptiINR currently searches for hyperparameter sets on individual, specific target signals rather than learning a generalizable configuration or weight initialization that transfers zero-shot across an entire distribution of signals.

Summary updated

Abstract

Implicit Neural Representations (INRs) have emerged as a transformative paradigm in signal processing and computer vision, excelling in tasks from image reconstruction to 3D shape modeling. Yet their effectiveness is fundamentally limited by the absence of principled strategies for optimal configuration—spanning activation selection, initialization scales, layer-wise adaptation, and their intricate interdependencies. These choices dictate performance, stability, and generalization, but current practice relies on ad-hoc heuristics, brute-force grid searches, or task-specific tuning, often leading to inconsistent results across modalities. We introduce OptiINR, the first unified framework that formulates INR configuration as a rigorous optimization problem. Leveraging Bayesian optimization, OptiINR efficiently explores the joint space of discrete activation families—sinusoidal (SIREN), wavelet-based (WIRE), variable-periodic (FINER)—and continuous initialization parameters. This systematic approach replaces fragmented manual tuning with a coherent, data-driven optimization process. By delivering globally optimal configurations, OptiINR establishes a principled foundation for INR design, consistently maximizing performance across diverse signal processing applications.
BibTeX
@misc{chen2025beyond,
  author = {Sipeng Chen and Yan Zhang and Shibo Li},
  title = {{Beyond Heuristics: Globally Optimal Configuration of Implicit Neural Representations}},
  year = {2025},
  eprint = {2509.23139},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23139},
}

Loading related content...

Figure