Weight Agnostic Neural Networks

Adam Gaier, David Ha

Introduction

In modern deep learning, the standard pipeline is well-established: select a human-designed architecture (like ResNet or a Transformer) and then expend massive computational resources finding the "perfect" weights via gradient descent. This paradigm raises a fundamental question: How much of a network's intelligence comes from its weights, and how much is inherent in its structure?

This paper challenges the conventional wisdom that neural network performance is primarily a product of weight optimization. By evolving network topologies that perform tasks using a single shared weight value sampled from a wide range, the authors demonstrate that architecture alone can encode complex solutions. This concept mirrors the innate "precocial" behaviors seen in newborn animals, suggesting that our current focus on weight optimization might be blinding us to the power of inductive biases—the inherent capabilities of a specific network layout.

Searching for Weight Agnostic Neural Networks (WANNs)

To explore this, the authors propose a search method for Weight Agnostic Neural Networks (WANNs). Instead of training weights, they evolve the network's topology using a process inspired by the NEAT (NeuroEvolution of Augmenting Topologies) algorithm.

The key innovations in their approach include:

  1. Weight Sharing & Sampling: During evaluation, every connection in a candidate network is assigned the exact same weight value. The network is tested over a range of these shared values (e.g., $[-2, -1, -0.5, +0.5, +1, +2]$).
  2. Performance Ranking: Architectures are ranked based on their mean performance across all sampled weight values and their simplicity (number of connections). This dual-objective ensures the found structures are both robust and minimal.
  3. Topological Evolution: The search starts with a minimal, partially connected network. It evolves by applying three operators: inserting a node (splitting a connection), adding a connection between unconnected nodes, or changing the activation function of a hidden node.

Domains and Key Insights

The researchers evaluated WANNs across two distinct domains: Continuous Control Reinforcement Learning (CartPoleSwingUp, BipedalWalker-v2, CarRacing-v0) and Supervised Learning (MNIST).

BipedalWalker WANN Architecture

The experiments revealed several striking insights about how architecture acts as an inductive bias:

  • Architecture as Computation: Specific arrangements of nodes and activation functions can "calculate" solutions directly. For example, in the CartPole task, the network evolves "inverters" on position inputs to create a centering force, effectively hard-coding a control loop into the geometry of the graph itself.
  • The Power of Diverse Activations: Unlike standard networks that rely heavily on ReLUs, WANNs make extensive use of diverse activation functions (Gaussian, Sinusoid, Step). This allows them to encode complex, non-linear relationships with far fewer nodes.
  • Minimalism: Because the search rewards simplicity, the resulting networks are incredibly small. The Bipedal Walker environment, for instance, was solved with just 210 connections—an order of magnitude fewer than standard deep RL baselines.
CarRacing WANN Architecture
  • Consistency over Precision: WANNs do not need "perfect" weights; they just need weights that are consistent in sign and roughly the right magnitude. This suggests that the relationships between inputs and outputs are more important than the exact strength of those relationships.

Key Results

The performance of these weight-agnostic architectures is surprisingly competitive:

  • MNIST: A WANN achieved ~92% accuracy using random weights. While not state-of-the-art, this is remarkable considering a standard linear classifier with random weights would perform at chance (~10%).
  • Continuous Control: WANNs successfully solved CartPole and BipedalWalker without any weight training. When the single shared weight was "tuned" (picked from the best of the sampled range), performance often rivaled traditionally trained networks.
  • Weight Tolerance: WANNs remained functional across a wide range of weight values. In contrast, fixed-topology networks (like standard MLPs) immediately failed if their weights were shifted even slightly from their trained values.

Limitations and Future Work

While groundbreaking, the WANN approach has limitations. The current evolutionary search is computationally expensive and operates in a discrete space. Moving toward differentiable architecture search could scale the method significantly. Furthermore, while WANNs are impressive, they still lag behind the absolute peak performance of large, weight-optimized models (e.g., 92% vs 99%+ on MNIST).

The current work also focuses strictly on feed-forward networks; incorporating recurrent connections for memory would be a logical and powerful next step. Ultimately, the goal isn't necessarily to replace weight training, but to find better "starting points." A WANN could serve as a highly capable "innate" structure that is then fine-tuned through learning to reach superhuman performance, bridging the gap between nature (architecture) and nurture (weight optimization).

Summary updated

Abstract

Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task. We propose a search method for neural network architectures that can already perform a task without any explicit weight training. To evaluate these networks, we populate the connections with a single shared weight parameter sampled from a uniform random distribution, and measure the expected performance. We demonstrate that our method can find minimal neural network architectures that can perform several reinforcement learning tasks without weight training. On a supervised learning domain, we find network architectures that achieve much higher than chance accuracy on MNIST using random weights.
BibTeX
@inproceedings{gaier2019weight,
  author = {Adam Gaier and David Ha},
  title = {{Weight Agnostic Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=E4cflu85Ez},
}
Figure