Neural Networks Trained to Solve Differential Equations Learn General Representations

Martin Magill, Faisal Z. Qureshi, Hendrick W. de HaanNeurIPS 2018

Introduction

In computer vision, it is well-established that the first layers of a convolutional neural network (CNN) almost always learn universal features, like edge detectors (Gabor filters), regardless of the specific classification task. This phenomenon is known as generality. However, it remained an open question whether this same principle applies to Differential Equation Neural Networks (DENNs)—networks specifically trained to solve complex physics problems.

Traditionally, generality is measured through "transfer learning" experiments: freezing a layer trained on Task A and evaluating its performance on Task B. While effective, this approach is computationally expensive and slow, particularly when studying a continuously-parametrized set of tasks (e.g., solving an equation as a physical source moves smoothly across a room). To address this, researchers have demonstrated a faster, mathematical way to quantify how "general" or "specific" a neural network layer truly is, revealing that DENNs also develop universal "generalized coordinates" in their early layers.

Methods and Datasets

Instead of relying on slow transfer learning protocols, the authors utilized Singular Vector Canonical Correlation Analysis (SVCCA). SVCCA compares the activation patterns of two layers to measure how much information they share. Because SVCCA is a linear algebraic method, it is several orders of magnitude faster than retraining networks.

The authors define two key metrics:

  • Reproducibility: How consistently a layer learns the same representation across different random initializations.
  • Specificity: How much a layer's representation changes when the underlying problem's parameters change. If different networks trained on different tasks learn the same internal representations, those representations are considered "general."

The primary dataset is a family of Boundary Value Problems (BVPs) based on the Parametrized Poisson Equation. The networks solve for the electric potential in a 2D square where a localized charge moves along the x-axis (position $x'$). This creates a continuous range of tasks. MNIST was also used as a secondary benchmark to validate intrinsic dimensionality and reproducibility metrics against existing literature.

Key Insights: Generality vs. Depth

The most striking finding is the discovery of a "Coordinate Map." The first hidden layers of these networks learn generalized coordinates. Rather than learning the specific solution to the physics problem, these early layers represent the geometry of the domain—identifying the four corners, the four walls, and the center.

SVCCA similarity heatmaps across layers

Just like in image recognition, generality is high in early layers and drops off as the network gets deeper. The first layer acts as a "universal toolkit" for the domain geometry, while the fourth layer acts as a "specialist" tuned to the specific location of the charge. Furthermore, the width of the network plays a role: narrow networks learn compressed representations, while wider networks converge toward a stable "intrinsic dimensionality" (about 9 dimensions for the first layer in their Poisson experiments).

Because differential equations are mathematically well-understood, physics serves as an excellent testbed. Unlike "black box" image data, researchers can interpret exactly what the neurons are doing (e.g., measuring distance from the top wall).

Key Results

The efficiency gains from this approach were massive. The authors computed all SVCCA specificities for the paper in about 15 minutes, whereas equivalent transfer learning experiments took hundreds of hours. Importantly, the SVCCA results showed excellent agreement with traditional transfer learning methods, validating SVCCA as a reliable proxy for measuring generality.

SVCCA similarity vs difference in task parameter

In a 4-layer network, the first two layers were found to be highly general across all problem variations (similarity remains flat regardless of the difference in the task parameter $x'$). The third layer's generality depended heavily on network width, and the fourth layer was consistently highly specific to the task (similarity drops as the difference in $x'$ increases).

Limitations and Future Work

While powerful, the SVCCA method has limitations. The authors noted that it can become numerically unstable when many neurons are highly correlated, making it difficult to sort the principal components of a layer. Additionally, in very wide networks, training sometimes hit "broad minima" and stopped before reaching the most general representations, suggesting that training protocols might need adjustment for massive networks.

Future work could expand beyond the simple square domain used in this study. Exploring "deformable domains" could reveal whether these networks can learn even more complex topological representations.

Summary updated

BibTeX
@inproceedings{magill2018neural,
  author = {Martin Magill and Faisal Z. Qureshi and Hendrick W. de Haan},
  title = {{Neural Networks Trained to Solve Differential Equations Learn General Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
}
Figure