Why Cannot Neural Networks Master Extrapolation? Insights from Physical Laws

TL;DR

While deep learning models excel at interpolating within their training data, they notoriously fail to extrapolate outside of it, often getting beaten by simple linear baselines. This paper reveals that standard neural networks with saturating activations are mathematically doomed to flatten into constants outside their training domain. By introducing the concept of "structural variability" derived from physical laws, the authors propose a simple architectural fix—combining subnetworks of varying depths—that significantly boosts extrapolation performance.

What Problem Does This Paper Address?

Why can a physics equation written centuries ago perfectly predict the behavior of a turbulent fluid it has never "seen," while a billion-parameter foundation model fails to forecast a basic time series just a few steps past its training window?

This paper tackles the extrapolation gap in deep learning. Standard statistical learning theory focuses on generalization—performing well on unseen data drawn from the same distribution as the training set. Extrapolation, however, requires predicting behavior completely outside the training domain. The authors seek to formalize the fundamental structural differences between neural networks and physical laws to understand why deep learning models fail so catastrophically at long-range forecasting.

Methods

The authors bridge the gap between machine learning and physics by analyzing models through the lens of differential algebra.

1. Differential Annihilators: The authors prove that neural networks (specifically MLPs with Tanh or Sigmoid activations) satisfy explicit, autonomous polynomial ordinary differential equations (ODEs). They call the minimal ODE satisfied by a network its differential annihilator.
2. Measuring Structural Variability: In physics, equations like the Navier-Stokes equation represent flexible data-generating processes capable of capturing vastly different regimes (like laminar vs. turbulent flow). The authors propose measuring a model's "structural variability" based on the order and algebraic properties of the ODEs its building blocks can satisfy.
3. The Linearly Combined Neural Network (LCNN): To fix the lack of structural variability in standard architectures, the authors introduce a deceptively simple modification. Instead of a standard MLP, they train a model that is a linear combination of multiple subnetworks of varying depths. This simple change forces the overall network to possess a diverse set of differential annihilator orders.

Datasets

The authors evaluated their framework on two types of data:

• Synthetic Regressions: Elementary and complex functions including Sine, Complex Periodic, Quadratic, and Hyperbolic Tangent ($tanh$) functions.
• Real-World Benchmark: The ETTH (Electricity Transformer Temperature) dataset, a standard benchmark for long-horizon time-series forecasting.

Key Insights and Ideas

• The Constant Trap: The most striking theoretical finding is that standard MLPs with Tanh or Sigmoid activations are mathematically guaranteed to converge exponentially fast to a constant value as they move away from the training domain boundary. This explains the common phenomenon where neural network predictions flatline during long-range forecasting.
• Occam's Razor via ODEs: Instead of traditional complexity metrics (like norms or parameter counts), the authors measure a function's complexity by the number of bits required to encode its corresponding polynomial ODE. Simpler ODEs represent simpler, more fundamental hypotheses that are inherently better suited for extrapolation.
• Variability Deficit: If a model's internal building blocks all satisfy the exact same differential operator, the model suffers from a "variability deficit." This deficit mathematically prevents it from spanning different qualitative regimes, leading to guaranteed extrapolation failure.

Key Results

The proposed LCNN architecture yielded massive performance gains in extrapolation scenarios:

Synthetic Tracking

On a simple Sine wave extrapolation task, a standard MLP completely flatlines outside the training boundaries, whereas the proposed LCNN successfully captures the continuing wave dynamics.

Quantitatively, the Mean Squared Error (MSE) dropped sharply across multiple synthetic functions:

• Sin: Standard MLP MSE of 1.172 vs. Proposed Net MSE of 0.082
• Complex Periodic: Standard MLP MSE of 1.592 vs. Proposed Net MSE of 0.416

Real-World Performance (ETTH Dataset)

When tested on electricity data across various extrapolation window sizes, the LCNN consistently outperformed the standard network architecture. For example, at a narrow extrapolation window (0.79), the MSE dropped from 15.00 (Standard) to 11.53 (Proposed).

Limitations and Future Work

While the theoretical insights are profound, the architectural solution presented is a "minimal first step." The authors demonstrate this fix using relatively small Multi-Layer Perceptrons (MLPs) with Sigmoid/Tanh activations.

The paper does not directly implement or test this architectural change on massive, state-of-the-art Transformer-based Foundation Models, nor does it explore non-saturating activation functions like ReLU in the same mathematical depth. The authors note that a promising future direction lies in completely merging symbolic models with over-parameterized deep learning architectures to inherently maximize structural variability.

@misc{dakhmouche2025why,
  author = {Ramzi Dakhmouche and Hossein Gorji},
  title = {{Why Cannot Neural Networks Master Extrapolation? Insights from Physical Laws}},
  year = {2025},
  eprint = {2510.04102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.04102},
}