Abstract: In machine learning, a field addressing the extraction of information and structure from finite data with the means of computer science and mathematics, maps from finite-dimensional spaces of data or computations into spaces of higher, or infinite dimensionality are a central theme. The workshop brought together researchers with diverse viewpoints to discuss how different theoretical sub-communities within the field treat the resulting ill-posed operations, and what kind of features of algorithms and models can emerge as a result.
Introduction: The Theory-Practice Gap
Modern machine learning is currently defined by a significant "theory-practice gap." While deep learning—particularly generative models and transformers—has achieved unprecedented empirical success, traditional theoretical frameworks struggle to explain why. Standard statistical learning theory suggests that as models become overparametrized (possessing far more parameters than data points), they should succumb to overfitting. Yet, modern architectures do the opposite, generalizing with remarkable efficiency.
Oberwolfach Report No. 4/2025 synthesizes a high-level workshop aimed at reconciling these successes with rigorous mathematics, focusing on the mechanisms of generalization, the quantification of uncertainty, and the transition from correlation-based models to causal inference.
The Mechanics of Generalization
A central theme of the workshop was the "Mystery of Generalization." Researchers explored several frameworks to understand why overparametrized models do not simply memorize training sets:
- Smoothness and "Spiky-Smooth" Estimators: A surprising insight is that smoothness, rather than just dimensionality, governs "benign overfitting." In fixed dimensions, overfitting can actually be beneficial if the estimator's derivatives are large enough to handle noise while maintaining a smooth underlying structure.
- Infinite-Width Theory: By utilizing Neural Tangent Kernels (NTK) and infinite-width limits, researchers are beginning to analyze the generalization properties of neural operators in fixed dimensions.
- Hyper-Kernel Ridge Regression (H-KRR): This method was shown to learn data representations similarly to neural networks, effectively escaping the "curse of dimensionality" that typically plagues standard kernel methods.
Uncertainty and Probabilistic Numerics
As machine learning models are integrated into critical infrastructure, "point estimates" are no longer sufficient. The workshop highlighted Probabilistic Numerics, an emerging field that treats computational errors—such as those from optimization or discretization—as sources of uncertainty that can be modeled with probability measures.
Key advancements in this area include:
- Scaling Gaussian Processes (GPs): New "computation-aware" GPs can now process up to 1.8 million datapoints on a single GPU, providing robust uncertainty quantification at scale.
- LUNO Framework: The Linearization-based Uncertainty for Neural Operators (LUNO) method allows for post-hoc Bayesian uncertainty quantification in pre-trained models with minimal computational overhead, requiring no retraining.
Architectural Insights: Transformers and SSMs
The workshop applied advanced mathematical tools to understand specific architectural successes:
- Rough Path Theory: This was used to study State-Space Models (SSMs) like Mamba, identifying that "gating mechanisms" are the crucial architectural choice driving their performance.
- Transformers as Causal Selectors: Analysis revealed "Selective Induction Heads" within transformers—specific circuits that allow the model to dynamically select the correct causal structure from a given context, rather than relying on a fixed Markovian structure.
- Associative Memory: Weight matrices in transformers can be viewed as associative memories; overparametrization increases their storage capacity and facilitates faster factual recall.
Causality and Optimization
Moving beyond pattern recognition toward causal understanding remains a primary goal. The workshop discussed Causal Representation Learning, which seeks to identify invariant components in data to allow models to generalize to "out-of-distribution" scenarios. A key theoretical contribution involved the Causal de Finetti theorem, suggesting that "exchangeable" data (where order is irrelevant but samples are dependent) contains richer information for breaking traditional limitations in causal discovery.
On the optimization front, the success of the Adam optimizer over standard SGD was attributed to its ability to handle the heavy-tailed class imbalances found in natural language. By normalizing gradients, Adam counteracts the ill-conditioning caused by infrequent tokens.
Future Directions and Limitations
Despite these advances, several frontiers remain:
- Scaling Theory to Depth: Most formal results currently apply to simpler model classes. Understanding how these mechanisms scale to complex, multi-layer production architectures is a major challenge.
- Causal Variable Discovery: Current models often require causal variables to be specified a priori. Future work must focus on discovering these variables directly from raw, high-dimensional data.
- The Educational Gap: There is a growing tension between teaching the foundational mathematical aspects of ML and the market demand for engineering "know-how."
In conclusion, the path toward reliable and interpretable AI lies in bridging these diverse mathematical viewpoints—from rough paths to probabilistic numerics—to create a unified theory of high-dimensional learning.
Summary updated