Grid-Functioned Neural Networks

Javier Dehesa, Andrew Vidler, Julian A. Padget, Christof LutterothICML 2021

Standard neural networks often rely on a "one-size-fits-all" approach, attempting to learn a single global function to map inputs to outputs. Grid-Functioned Neural Networks (GFNN) introduce a clever architectural shift: structuring network parameters into a multidimensional grid of experts. By interpolating these expert weights on-the-fly, GFNNs achieve higher precision and lower computational costs in complex regression tasks.

Introduction

GFNNs replace static network weights with a multidimensional grid of "expert" parameters that are dynamically blended based on input features. This allows the model to specialize in different subspaces of a problem while maintaining smooth, continuous behavior. Ultimately, GFNNs outperform standard Multilayer Perceptrons (MLPs) and state-of-the-art animation models in both accuracy and inference speed.

The Problem with Global Functions

Standard neural networks, such as MLPs, often act as opaque containers that attempt to learn a single global function for an entire domain. This frequently leads to an "averaging" effect, where the model struggles to capture fine-grained local patterns or "minor" features that end up overshadowed by "gross" features.

While "Ensembles of Experts" exist to solve this, they are often unstructured, making them hard to train and prone to discontinuities. Previous structured approaches, like Phase-Functioned Neural Networks (PFNN), were limited to a single dimension of specialization (e.g., the phase of a walking cycle). GFNN generalizes this concept to any number of dimensions, providing a flexible, continuous, and efficient way to handle complex, multi-faceted data.

Methodology: The Expert Grid

The core idea of a GFNN is to make the network's weights a function of the input itself.

Architecture of a Grid-Functioned Neural Network
  1. The Expert Grid: The user defines $K$ features of the input (e.g., speed, direction, or spatial coordinates) that act as "specialization dimensions." These dimensions form a multidimensional grid where each vertex contains a full set of parameters (weights and biases) for a base MLP.
  2. Multidimensional Interpolation: When an input is processed, the model identifies the surrounding "expert" parameters in the grid. It uses multidimensional cubic spline interpolation (specifically Catmull-Rom splines) to blend these experts into a single set of parameters tailored for that specific input.
  3. Smoothness ($C^1$ Continuity): Because it uses cubic splines, the transition between experts is mathematically smooth. This is critical for applications like 3D animation, where sudden jumps in parameters would cause visible "glitches" or "pops."
  4. Inference Flexibility: The architecture provides three ways to run the model at inference time:
  5. Cubic: Highest accuracy, highest computational cost.
  6. Linear: A faster approximation using linear interpolation.
  7. Stepwise: The fastest method, simply picking the nearest expert (useful for highly resource-constrained environments).

Experimental Setup

The authors evaluated GFNN across three distinct scenarios:

  • Synthetic Regression: Fitting the Rosenbrock and Ackley functions. These are standard mathematical benchmarks used to test a model's ability to navigate complex, "peaky" surfaces.
  • Revised Ackley: A high-frequency version of the Ackley function designed to test how the model scales with more training data and larger grids.
  • Quadruped Locomotion: A real-world 3D animation dataset involving a four-legged character. The grid dimensions here were defined by the character's velocity and future trajectory angle.

Key Results and Insights

GFNNs demonstrate a clear advantage in capturing high-frequency details that standard MLPs simply smooth over.

Surface fitting comparison between standard MLP and GFNNs of varying grid sizes.
  • Superior Surface Fitting: On the complex Ackley function, GFNNs reduced mean error by 60% to 95% compared to a standard MLP with a similar number of parameters. As the grid density increased (e.g., from $3 \times 3$ to $7 \times 7$), the accuracy improved dramatically.
  • Animation Accuracy: In the quadruped case study, GFNN was significantly better at adhering to requested control commands (like specific speeds) than the previous state-of-the-art MANN (Mode-Adaptive Neural Network).
  • The Efficiency Paradox: One might assume a grid of experts would be slower. However, because each expert only needs to solve a small "sub-problem," the individual networks can be much smaller. A GFNN with a large grid of tiny experts is often faster (requiring fewer total FLOPs) than one giant MLP trying to learn everything.
  • Speedup: Using linear interpolation, the GFNN was ~30% faster than the MANN model while maintaining comparable visual quality in a user study.
  • Explicit vs. Emergent Specialization: Unlike models that try to learn how to partition a space (which can be unstable), GFNN lets the designer define the axes of specialization. This makes the model much more predictable and easier to initialize.

Limitations and Future Directions

While powerful, GFNNs are not without limitations. The most prominent is the Curse of Dimensionality: the number of experts in the grid grows exponentially with the number of specialization dimensions ($N^K$). While the authors note that the number of experts participating in a single inference remains constant (only 4 per dimension for cubic interpolation), the memory footprint for very high-dimensional grids could become prohibitive.

Additionally, the user must currently manually choose which input features define the grid. Future work could explore automated ways to identify these optimal dimensions. Finally, while this paper focuses on MLPs, the "grid-functioned" concept could theoretically be applied to convolutional or recurrent layers, opening an exciting new area for architectural research.

Summary updated

BibTeX
@inproceedings{dehesa2021grid,
  author = {Javier Dehesa and Andrew Vidler and Julian A. Padget and Christof Lutteroth},
  title = {{Grid-Functioned Neural Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure