Introduction
Despite the transformative success of deep learning, a comprehensive theoretical foundation remains elusive. We currently lack a "white-box" map to navigate the complex landscape of modern neural networks, where sophisticated heuristics often outpace our mathematical understanding. To bridge this gap, we propose Neurashed, a phenomenological model designed to capture the essential mechanics of deep learning—hierarchy, iterative optimization, and data compression—without the overwhelming complexity of billions of parameters.
The Neurashed Framework
Neurashed is a graphical model that imitates the training process of a neural network by representing information flow through a hierarchical structure. Unlike standard networks where nodes are abstract mathematical functions, each node in Neurashed represents a specific feature (e.g., a "pointy shape" or a "panda head").
The model operates on Boolean firing rules: a high-level feature node fires only if its constituent low-level feature nodes in the previous layer fire. This "watershed" analogy describes how low-level information streams meet to form complex concepts.
Mechanisms of Learning
The training dynamics of Neurashed are governed by an iterative, Hebbian-inspired rule: "cells that fire together, wire together." During each training step:
- Amplification: If a node fires, its amplification factor ($\lambda_F$) increases, strengthening its pathway.
- Regularization: If a node remains inactive, its factor decreases, mimicking weight decay or synaptic pruning.
This competition between nodes naturally leads to implicit regularization. In small-batch training, only a few pathways fire at once, forcing the model to learn sparse, efficient representations. Over time, dominant features "dwarf" noisier ones, leading to the emergence of an information bottleneck.
Empirical Validation and Results
Neurashed effectively explains several empirical patterns observed in deep learning, most notably the Information Bottleneck (IB) and local elasticity.
Information Bottleneck
Simulations using Neurashed successfully reproduce the two-phase learning process: an initial "fitting" phase where mutual information (MI) with the label increases, followed by a "compression" phase where MI with the input decreases as the model discards irrelevant details.
Local Elasticity
To validate the concept of local elasticity—the idea that updates to one class primarily affect similar classes—we benchmarked against a VGG-19 model trained on ImageNet (specifically brown bears, leopards, and tigers). Neurashed mathematically predicts that as network depth increases, updates become increasingly "local." This explains why deep networks are robust; training on a "leopard" boosts "tiger" recognition (shared pathways) while leaving "brown bear" logic largely untouched.
Discussion
While Neurashed provides a powerful conceptual tool, it currently utilizes fixed feature nodes and connections. In real-world neural networks, these features are learned dynamically. Future work will focus on introducing adaptive connections and stochastic update rules to more closely mirror the noise inherent in stochastic gradient descent. By providing a transparent map of these dynamics, Neurashed serves as a stepping stone toward a more rigorous theory of deep learning.
Summary updated