Neurashed: A Phenomenological Model for Imitating Deep Learning Training

Weijie J. Su

Introduction

Despite the transformative success of deep learning, a comprehensive theoretical foundation remains elusive. We currently lack a "white-box" map to navigate the complex landscape of modern neural networks, where sophisticated heuristics often outpace our mathematical understanding. To bridge this gap, we propose Neurashed, a phenomenological model designed to capture the essential mechanics of deep learning—hierarchy, iterative optimization, and data compression—without the overwhelming complexity of billions of parameters.

The Neurashed Framework

Neurashed is a graphical model that imitates the training process of a neural network by representing information flow through a hierarchical structure. Unlike standard networks where nodes are abstract mathematical functions, each node in Neurashed represents a specific feature (e.g., a "pointy shape" or a "panda head").

Hierarchical feature activation in Neurashed for different input classes.

The model operates on Boolean firing rules: a high-level feature node fires only if its constituent low-level feature nodes in the previous layer fire. This "watershed" analogy describes how low-level information streams meet to form complex concepts.

Mechanisms of Learning

The training dynamics of Neurashed are governed by an iterative, Hebbian-inspired rule: "cells that fire together, wire together." During each training step:

  1. Amplification: If a node fires, its amplification factor ($\lambda_F$) increases, strengthening its pathway.
  2. Regularization: If a node remains inactive, its factor decreases, mimicking weight decay or synaptic pruning.

This competition between nodes naturally leads to implicit regularization. In small-batch training, only a few pathways fire at once, forcing the model to learn sparse, efficient representations. Over time, dominant features "dwarf" noisier ones, leading to the emergence of an information bottleneck.

Empirical Validation and Results

Neurashed effectively explains several empirical patterns observed in deep learning, most notably the Information Bottleneck (IB) and local elasticity.

Information Bottleneck

Simulations using Neurashed successfully reproduce the two-phase learning process: an initial "fitting" phase where mutual information (MI) with the label increases, followed by a "compression" phase where MI with the input decreases as the model discards irrelevant details.

Information plane showing the fitting and compression phases during training.

Local Elasticity

To validate the concept of local elasticity—the idea that updates to one class primarily affect similar classes—we benchmarked against a VGG-19 model trained on ImageNet (specifically brown bears, leopards, and tigers). Neurashed mathematically predicts that as network depth increases, updates become increasingly "local." This explains why deep networks are robust; training on a "leopard" boosts "tiger" recognition (shared pathways) while leaving "brown bear" logic largely untouched.

Discussion

While Neurashed provides a powerful conceptual tool, it currently utilizes fixed feature nodes and connections. In real-world neural networks, these features are learned dynamically. Future work will focus on introducing adaptive connections and stochastic update rules to more closely mirror the noise inherent in stochastic gradient descent. By providing a transparent map of these dynamics, Neurashed serves as a stepping stone toward a more rigorous theory of deep learning.

Summary updated

Abstract

To advance deep learning methodologies in the next decade, a theoretical framework for reasoning about modern neural networks is needed. While efforts are increasing toward demystifying why deep learning is so effective, a comprehensive picture remains lacking, suggesting that a better theory is possible. We argue that a future deep learning theory should inherit three characteristics: a \textit{hierarchically} structured network architecture, parameters \textit{iteratively} optimized using stochastic gradient-based methods, and information from the data that evolves \textit{compressively}. As an instantiation, we integrate these characteristics into a graphical model called \textit{neurashed}. This model effectively explains some common empirical patterns in deep learning. In particular, neurashed enables insights into implicit regularization, information bottleneck, and local elasticity. Finally, we discuss how neurashed can guide the development of deep learning theories.
BibTeX
@article{su2024neurashed,
  author = {Weijie J. Su},
  title = {{Neurashed: A Phenomenological Model for Imitating Deep Learning Training}},
  journal = {Science China Information Sciences},
  year = {2024},
  url = {https://openreview.net/forum?id=VfUa9s2CNW},
}
Figure