A Renormalization Group Framework for Scale-Invariant Feature Learning in Deep Neural Networks (Student Abstract)

Sarah LiawAAAI 2025 · DOI

Introduction

In the field of deep learning, the hierarchical nature of feature extraction—where early layers detect simple edges and deeper layers identify complex objects—is well-documented but lacks a rigorous mathematical foundation. This research, presented as a student abstract at AAAI-25, proposes a novel framework based on Renormalization Group (RG) theory from statistical physics to formalize this process. By treating neural network layers as "coarse-graining" operations, the framework aims to optimize how models learn scale-invariant features.

The Challenge of Scale Invariance

A persistent challenge in computer vision and signal processing is scale invariance: the ability of a model to recognize a pattern regardless of its size. While current techniques rely heavily on data augmentation (e.g., training on various image scales), this approach does not inherently encode scale invariance into the network's architecture. The proposed RG framework seeks to bridge this gap by using tools from statistical physics to analyze and control how information is filtered and transformed across network layers.

The RG Framework for Deep Learning

The core of the proposal lies in unifying deep learning operations with RG transformations. This is achieved through three primary mathematical innovations:

1. Scale-Aware Activation Functions

The author introduces a class of activation functions, denoted as $\sigma_s$, designed to satisfy specific scaling properties. These functions ensure that if an input is scaled, the resulting output scales in a predictable, covariant manner, providing a mathematical guarantee of consistency across different levels of observation.

2. RG Flow Equations and Beta Functions

In physics, RG flows describe the evolution of a system's parameters as the scale changes. The author derives "beta functions" ($\beta(\theta)$) for neural network weights and biases. These functions quantify how parameters must evolve to maintain feature invariance, effectively mapping the learning process to a physical trajectory toward a stable state.

3. RG-Guided Training

Moving beyond standard gradient descent, the paper proposes a modified optimization rule: $$\theta \leftarrow \theta - \eta \nabla_\theta L - \epsilon \beta(\theta)$$ This update rule incorporates an "RG step" ($\epsilon \beta(\theta)$) that nudges the network parameters toward fixed points. At these fixed points, the features extracted by the network become perfectly invariant to scale changes, theoretically leading to more robust representations.

Theoretical Insights and Information Theory

The framework provides a formal mapping between neural network layers and RG coarse-graining. Just as RG in physics strips away microscopic details to study macroscopic phenomena—such as phase transitions in the Ising model—the proposed network layers filter out fine-grained noise to preserve essential large-scale structures.

The paper supports these claims with several key theorems:

  • Theorem 2: Derives the specific beta functions for weights and biases based on the scaling dimension ($\alpha$) of the activation function.
  • Theorem 4: Proves that the RG-guided training procedure converges to a fixed point while simultaneously minimizing the loss function.
  • Theorem 5: Uses mutual information to demonstrate that information is lost monotonically as it moves through layers, confirming the intuition that deep networks function as systematic coarse-graining filters.

Conclusion and Future Directions

While the current work is primarily theoretical, it offers a promising path toward more interpretable and scale-invariant models. By linking deep learning to the behavior of physical systems near phase transitions, the framework provides a new lens through which to view neural architecture design.

The author notes that future research will focus on empirical validation. Key objectives include testing the RG-guided training on standard benchmarks like ImageNet, applying the framework to time-series and wavelet analysis, and investigating the model's performance on out-of-distribution data.

Summary updated

BibTeX
@inproceedings{liaw2025renormalization,
  author = {Sarah Liaw},
  title = {{A Renormalization Group Framework for Scale-Invariant Feature Learning in Deep Neural Networks (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35269},
}
Figure