Abstract
Can general-purpose AI architectures go beyond prediction to discover the physical laws governing the universe? True intelligence relies on "world models" -- causal abstractions that allow an agent to not only predict future states but understand the underlying governing dynamics. While previous "AI Physicist" approaches have successfully recovered such laws, they typically rely on strong, domain-specific priors that effectively "bake in" the physics. Conversely, Vafa et al. recently showed that generic Transformers fail to acquire these world models, achieving high predictive accuracy without capturing the underlying physical laws. We bridge this gap by systematically introducing three minimal inductive biases. We show that ensuring spatial smoothness (by formulating prediction as continuous regression) and stability (by training with noisy contexts to mitigate error accumulation) enables generic Transformers to surpass prior failures and learn a coherent Keplerian world model, successfully fitting ellipses to planetary trajectories. However, true physical insight requires a third bias: temporal locality. By restricting the attention window to the immediate past -- imposing the simple assumption that future states depend only on the local state rather than a complex history -- we force the model to abandon curve-fitting and discover Newtonian force representations. Our results demonstrate that simple architectural choices determine whether an AI becomes a curve-fitter or a physicist, marking a critical step toward automated scientific discovery.
Introduction
The quest for an "AI Scientist" hinges on the ability of neural architectures to move beyond mere pattern recognition toward the discovery of fundamental physical laws. While Transformers have revolutionized natural language processing through next-token prediction, their capacity to learn causal "world models"—internal representations of the governing dynamics of a system—remains a subject of intense debate. Recent work by Vafa et al. (2025) highlighted a significant limitation: generic Transformers trained on planetary trajectories can predict future positions with high accuracy but fail to internalize the underlying Newtonian forces. Instead, they act as sophisticated curve-fitters, memorizing geometric patterns without grasping the causal mechanisms.
Methods: From Predictors to Physicists
The transition from a "curve-fitter" to a "physicist" does not require complex, domain-specific equations to be hard-coded into the model. Instead, the authors demonstrate that three minimal inductive biases can guide a generic Transformer toward discovering physical laws.
1. Spatial Smoothness
Standard tokenization treats nearby coordinates as independent discrete categories. This "smoothness killer" prevents the model from understanding that points close in physical space should have similar internal representations. By replacing discrete classification with continuous regression, the model is forced to learn a smooth spatial map where the distance in latent space reflects the distance in physical space.
2. Spatial Stability
Autoregressive models are prone to error accumulation, where small inaccuracies in early predictions lead to divergent trajectories. To mitigate this, the authors employ noisy context learning. By injecting Gaussian noise into the training inputs, the model learns to be self-correcting, effectively discovering how to map noisy states back onto the stable manifold of the physical system.
3. Temporal Locality
The most critical bias for discovering Newtonian physics is temporal locality. By restricting the attention window to only the most recent states (e.g., a context length of 2), the model is prevented from seeing the global "shape" of the trajectory. This constraint forces the architecture to discover local invariants—specifically, the gravitational force vectors—that determine the next state based solely on the current local configuration.
Key Insights and Results
The Context Length Phase Transition
The researchers found that the choice of context length triggers a "phase transition" in the model's internal representation. With a long context window, the Transformer behaves like Kepler, representing the orbit through global parameters like the semi-major axis and eccentricity. However, when the context is restricted to just two states, the model is forced to act like Newton. Linear probing of the latent embeddings reveals that the model discovers internal representations of gravitational force magnitude and direction with near-perfect accuracy ($R^2 \approx 0.999$).
Scaling Laws for Spatial Maps
The study derives a scaling law for spatial map quality, showing that $1 - R^2 \approx AD^{-\alpha_D} V^{\alpha_V}$. Crucially, they find that vocabulary size ($V$) matters more than model size. If the vocabulary is too large, the model struggles to "cover" all tokens during training, preventing a smooth spatial map from forming. Training data ($D$) must grow at least as fast as $V$ to maintain map quality.
Robustness vs. Insight
While the Newtonian (local) model provides deeper physical insight, the Keplerian (global) model is often more robust for long-term predictions in the presence of noise. This is because the Keplerian model relies on the stable geometry of the entire orbit, whereas the Newtonian model must integrate local forces, which can lead to faster error accumulation if the local dynamics are not perfectly captured.
Conclusion
This work provides a blueprint for building AI systems capable of scientific discovery. By architecturally constraining models to look for simple, local mechanisms rather than complex global histories, we can move from AI that merely predicts the future to AI that understands the laws governing it. Future research will focus on extending these principles to multi-scale phenomena and extracting explicit symbolic laws from these learned representations.
Summary updated