Patterns, predictions, and actions: A story about machine learning

Moritz Hardt, Benjamin Recht

Machine learning is frequently presented as a fragmented collection of algorithms and "black box" heuristics. In their work, Patterns, Predictions, and Actions, Moritz Hardt and Benjamin Recht offer a foundational re-framing of the field. They argue that modern AI is essentially a high-powered evolution of 1960s "pattern classification," and they seek to bridge the conceptual gap between static statistical prediction and real-world decision-making.

The cover of "Patterns, Predictions, and Actions" by Moritz Hardt and Benjamin Recht.

The Generalization Mystery and Stability

A central question in modern deep learning is why massive, overparameterized models generalize so well to new data instead of simply memorizing the training set. Traditional statistical theory suggests that as model complexity increases, the risk of overfitting grows (the classic U-shaped risk curve). However, the authors highlight the "Double Descent" phenomenon: increasing model complexity beyond the point of perfectly fitting the training data can actually improve test performance.

To explain this, the authors move away from traditional tools like VC-dimension and instead emphasize algorithmic stability. The core idea is that if a small change in the training data (e.g., changing a single data point) does not drastically alter the resulting model, that model is likely to generalize well. They derive stability bounds showing that the probability of error on new data is controlled by the ratio of the data's "diameter" to the "margin" of the separator, independent of the actual number of dimensions.

From Prediction to Action: The Causal Gap

One of the most significant contributions of this work is the distinction between prediction (identifying a pattern) and action (using that pattern to intervene in the world). A model might accurately predict that a person is likely to default on a loan based on historical data, but that does not mean the model understands the causal impact of granting or denying that loan.

To address this "Prediction-Action Gap," the authors integrate Structural Causal Models (SCMs) and Pearl’s "do-calculus." They use examples like Simpson’s Paradox—specifically the famous UC Berkeley admissions case—to demonstrate how purely observational data can lead to incorrect conclusions if causal factors, such as department choice, are not explicitly modeled. In the realm of Reinforcement Learning (RL), they advocate for a "predict then act" baseline known as certainty equivalence, which often proves more robust than complex "model-free" approaches.

Optimization as a Lens

The success of deep learning is framed not just through neural architectures, but through the lens of optimization. The authors argue that Stochastic Gradient Descent (SGD) possesses "implicit regularization" properties. Even though deep learning objectives are non-convex and theoretically "NP-hard," overparameterization creates a form of "hidden convexity" in the space of predictions. This allows SGD to find global minima efficiently, effectively bypassing the worst-case theoretical hurdles.

Datasets as Social Benchmarks

The work provides a historical deep dive into the role of benchmarks, from Highleyman’s 1959 data to MNIST and ImageNet. The authors argue that datasets are not neutral mathematical objects but historical artifacts and social contracts within the research community.

While benchmarks facilitate progress, they can lead to "adaptive overfitting," where the community collectively overfits to a specific test set over years of iteration. Furthermore, the authors point to the Fragile Families Challenge as a pivotal example of the limits of prediction. In this challenge, complex models using over 12,000 variables failed to significantly outperform simple linear models using only four variables, highlighting that some human systems are inherently unpredictable regardless of algorithmic sophistication.

Conclusion: Beyond Data

Hardt and Recht conclude that "data alone is not enough." While pattern classification has reached extraordinary heights, future progress in high-stakes areas like healthcare and public policy requires re-integrating domain-specific causal knowledge. The challenge for the next generation of machine learning is to move beyond observing patterns to safely and ethically taking actions in a changing world.

Summary updated

Abstract

This graduate textbook on machine learning tells a story of how patterns in data support predictions and consequential actions. Starting with the foundations of decision making, we cover representation, optimization, and generalization as the constituents of supervised learning. A chapter on datasets as benchmarks examines their histories and scientific bases. Self-contained introductions to causality, the practice of causal inference, sequential decision making, and reinforcement learning equip the reader with concepts and tools to reason about actions and their consequences. Throughout, the text discusses historical context and societal impact. We invite readers from all backgrounds; some experience with probability, calculus, and linear algebra suffices.
BibTeX
@misc{hardt2021patterns,
  author = {Moritz Hardt and Benjamin Recht},
  title = {{Patterns, predictions, and actions: A story about machine learning}},
  year = {2021},
  eprint = {2102.05242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.05242},
}
Figure