Abstract
Implicit Neural Representations (INRs) have emerged as a powerful paradigm for representing continuous signals—such as images, audio, and 3D shapes—using coordinate-based neural networks. However, traditional INRs rely on global Multi-Layer Perceptrons (MLPs) that struggle with spectral bias and inefficient scaling. We present "Neural Experts," a Mixture of Experts (MoE) architecture specifically designed for INRs. By decomposing the signal domain into local, piece-wise continuous regions, Neural Experts achieve higher reconstruction fidelity and better parameter efficiency than global architectures. Our method introduces specialized manager conditioning and pretraining techniques to ensure stable routing and optimal expert utilization.
Introduction
Implicit Neural Representations (INRs) represent a shift from discrete, grid-based data structures to continuous functions. By mapping coordinates to signal values (e.g., $x, y \to RGB$), INRs provide a compact and resolution-independent format for complex data. Despite their success, standard MLPs face two primary challenges: global constraints and spectral bias. Because every parameter in a global MLP contributes to every part of the signal, optimization becomes increasingly difficult as signal complexity grows. Furthermore, MLPs naturally favor low-frequency functions, often smoothing over sharp boundaries and fine details.
To address these limitations, we introduce Neural Experts. This framework adopts a Mixture of Experts (MoE) approach, allowing the model to subdivide the input space and assign specialized sub-networks to specific local regions. This localization enables the representation of piece-wise continuous functions, which are better suited for real-world signals containing sharp edges and high-frequency transitions.
The Neural Experts Architecture
The Neural Experts framework consists of two primary components: a set of Experts and a Manager network.
1. The Experts
The experts are a collection of small, independent sub-networks ($N_{experts}$). Each expert is responsible for learning the signal representation within a specific sub-region of the input domain. By limiting the scope of each expert, the model can capture local nuances without being constrained by the global characteristics of the entire signal.
2. The Manager
The manager network acts as a router, determining which expert is best suited to handle a given input coordinate $x$. To ensure the manager makes informed decisions, we implement two critical innovations:
- Conditioning: The manager does not operate on raw coordinates alone. Instead, it is conditioned by concatenating its internal features with features from an expert encoder. This shared context allows the manager to align its routing logic with the experts' learned representations.
- Manager Pretraining: MoE models often suffer from "expert starvation," where a few experts are over-utilized while others remain untrained. We mitigate this by pretraining the manager to output a random uniform assignment across the domain. This forces the system to initially distribute the learning load evenly before refining the boundaries between experts.
Training concludes with a refinement phase where the manager's weights are frozen, allowing the experts to fine-tune their local representations without the instability of shifting assignments.
Experimental Evaluation
We evaluated Neural Experts across three distinct modalities to demonstrate its versatility and performance.
Image Reconstruction
Using the Kodak dataset, Neural Experts were paired with Sine activations (SIREN-based). The architecture achieved a Peak Signal-to-Noise Ratio (PSNR) of approximately 89, significantly outperforming the standard SIREN baseline (~57) and even a "Wider" SIREN variant (~77). This demonstrates the model's superior ability to capture high-frequency image details.
Audio Modeling
In tests involving raw waveforms—including classical music and speech—Neural Experts outperformed standard MLPs while utilizing 40% fewer parameters. Notably, in a "Two Speakers" scenario, the manager network exhibited emergent semantic behavior, naturally assigning different experts to different speakers without explicit supervision.
3D Surface Reconstruction
We tested the model on complex 3D shapes from the Stanford 3D scanning repository, such as the Thai Statue, using Signed Distance Functions (SDFs). Neural Experts captured intricate details in high-curvature areas (e.g., eyes and nostrils) that global MLPs typically fail to resolve. Quantitatively, the model achieved a Mean Trimap IoU of 0.8180, surpassing the 0.6662 achieved by a larger SIREN model, despite using fewer total parameters (1.3M vs 1.5M).
Discussion and Insights
Our findings suggest several key advantages of the MoE approach for INRs:
- Piece-wise Continuity: Real-world signals are rarely globally smooth. By learning piece-wise continuous functions, Neural Experts provide a more natural fit for data with sharp transitions.
- Horizontal vs. Vertical Scaling: Our experiments indicate that for coordinate-based tasks, increasing the number of experts (horizontal scaling) is more effective than increasing the depth of individual experts (vertical scaling).
- Inference Efficiency: Because only the selected expert is activated for any given coordinate, the model offers sparse computation, allowing for high-capacity representations with low inference-time costs.
Limitations and Future Work
While Neural Experts offer significant gains, training can be computationally intensive and "noisy" during the early stages as the manager learns to partition the space. Additionally, when using SoftPlus activations, the model still faces some difficulty in fitting extremely low-frequency components. Future research will explore the integration of MoE-INRs with generative diffusion models and the application of sharding techniques to scale the architecture for city-scale 3D reconstructions.
Summary updated