Intelligence, physics and information - the tradeoff between accuracy and simplicity in machine learning

Tailin WuarXiv 2020 · OpenReview

Abstract

Modern deep learning models are often characterized as powerful "black boxes" that prioritize memorization over conceptual understanding. This research explores the fundamental tradeoff between task accuracy and model simplicity to develop more intelligent, generalizable machine learning systems. By integrating principles from physics and information theory, this work introduces novel methodologies for symbolic law discovery, representation learning analysis, causal inference, and robust training on noisy datasets.

Introduction: The Limits of Modern Deep Learning

While deep learning has achieved remarkable success across various domains, it faces significant hurdles in generalization and interpretability. Standard models often require massive datasets to learn simple physical dynamics and remain highly sensitive to noisy or mislabeled data. To bridge the gap between human-like reasoning and machine intelligence, this research posits that learning should be viewed as an optimization problem balancing predictive power against description length—a modern application of Occam’s Razor.

The AI Physicist: Discovering Symbolic Laws

One of the primary contributions of this work is the "AI Physicist," an agent designed to learn interpretable, symbolic laws of physics from limited observations. Unlike monolithic neural networks, the AI Physicist employs a "divide-and-conquer" strategy, maintaining a library of small, specialized neural networks for different physical domains.

To ensure the learned models are interpretable, the agent applies the principle of Minimum Description Length (MDL). This process "snaps" learned weights into simple rational numbers and symbolic formulas, effectively recovering exact physical parameters (such as the gravitational constant $g$). In experimental evaluations using 2D simulations of harmonic oscillators and chaotic double pendulums, the AI Physicist achieved mean-squared prediction errors a billion times smaller than standard feedforward networks and successfully solved over 90% of mystery environments.

Representation Learning as a Phase Transition

Drawing on the Information Bottleneck (IB) framework, the research investigates how models transition from noise to signal. A key insight is that representation learning does not occur smoothly; rather, it happens in discrete "phase transitions."

Using second-order calculus of variations, the author mathematically predicts the exact hyperparameter thresholds ($\beta$) where a model begins to learn non-trivial representations. These transitions are determined by the dataset's "conspicuous subset"—the largest, most confident, and typical subset of examples. This theoretical framework was validated on standard benchmarks like MNIST and CIFAR-10, where the derived formulas accurately predicted the onset of class-specific learning.

Causal Discovery and Meta-Learning

Beyond physical laws and representation theory, the thesis addresses how agents can infer structure from time-series data and adapt to new tasks rapidly:

  • Minimum Predictive Information Regularization (MPIR): This method for causal discovery trains networks to predict future states while penalizing the mutual information between the input and the prediction. This forces the model to identify the underlying nonlinear causal relationships rather than relying on superficial correlations.
  • Meta-Learning Autoencoders (MeLA): To facilitate few-shot learning, the MeLA architecture uses an autoencoder to distill examples into a compact "model code." A meta-generative model then uses this code to instantly generate weights for a task-specific network. MeLA has demonstrated the ability to outperform state-of-the-art algorithms like MAML on physics prediction tasks with zero or minimal gradient steps.

Robustness via Rank Pruning

In real-world scenarios, training data is often plagued by mislabeled examples. The research introduces "Rank Pruning," a fast algorithm for learning with noisy labels. The core idea is that models do not need perfectly calibrated probabilities to filter noise; they only need to accurately rank examples by confidence. By estimating noise rates and pruning the most likely mislabeled data prior to training, Rank Pruning achieved state-of-the-art F1 scores. Notably, it maintained a 0.46% error rate on MNIST even when 50% of the positive examples were intentionally mislabeled.

Conclusion and Future Directions

This research demonstrates that the integration of physical principles and information-theoretic constraints leads to models that are not only more accurate but also more robust and interpretable. While challenges remain in modeling highly chaotic systems and scaling IB analysis to continuous high-dimensional data, the methodologies presented—from the AI Physicist to Rank Pruning—provide a roadmap for building unified intelligent agents. Future work will focus on integrating these modules into a single architecture capable of autonomous discovery and robust reasoning across diverse environments.

Summary updated

BibTeX
@misc{wu2020intelligence,
  author = {Tailin Wu},
  title = {{Intelligence, physics and information - the tradeoff between accuracy and simplicity in machine learning}},
  year = {2020},
  url = {https://openreview.net/forum?id=cJfGRUsrKh},
}
Figure