A Simple Model of Inference Scaling Laws

Noam LeviarXiv 2024 · OpenReview

Abstract

Neural scaling laws have garnered significant interest due to their ability to predict model performance as a function of increasing parameters, data, and compute. In this work, we propose a simple statistical ansatz based on memorization to study scaling laws in the context of inference, specifically how performance improves with multiple inference attempts. We explore the coverage, or pass@k metric, which measures the chance of success over repeated attempts and provide a motivation for the observed functional form of the inference scaling behavior of the coverage in large language models (LLMs) on reasoning tasks. We then define an "inference loss", which exhibits a power law decay as the number of trials increases, and connect this result with prompting costs. We further test our construction by conducting experiments on a simple generative model, and find that our predictions are in agreement with the empirical coverage curves in a controlled setting. Our simple framework sets the ground for incorporating inference scaling with other known scaling laws.

Introduction

In the field of Large Language Models (LLMs), scaling laws have traditionally focused on training—predicting how performance improves as parameters and training data increase. However, recent empirical evidence suggests that models also "scale" at inference time. When a model is given multiple attempts to solve a difficult problem, its performance—measured by the pass@k metric—improves predictably, provided an external verifier can identify the correct solution.

Despite the practical success of inference-time scaling, a theoretical foundation has been largely absent. This paper addresses this gap by introducing a simple mathematical framework to model how performance scales with the number of inference attempts, treating model failure as a statistical distribution of task difficulty.

The Memorizing Ansatz and Task Difficulty

The proposed framework relies on a "Memorizing Ansatz." Instead of attempting to model the complex internal reasoning of an LLM, the paper treats the model as a noisy retriever. In this view, the model has memorized its training data but may occasionally fail to retrieve the correct sequence of steps for a specific task.

To account for varying task difficulty, the author models failure probabilities using a Beta distribution. This allows the framework to represent a mix of "easy" tasks (low failure probability) and "hard" tasks (high failure probability). By averaging over this distribution, a closed-form expression for pass@k is derived.

A critical refinement in this model is the introduction of "Effective $k$" ($k_{eff}$). Recognizing that repeated attempts by the same model are not truly independent—as the model might be "stuck" in a specific incorrect reasoning path—the author incorporates a correction factor to account for correlated trials.

Inference Scaling Laws and the Power Law Decay

A major insight of this work is the definition of a new metric: "inference loss" ($\mathcal{L}_{inference}$). This metric measures the expected error across $k$ trials, serving as an inference-time analogue to the standard test loss used during training.

The analysis demonstrates that inference loss follows a power-law decay ($k^{-\beta}$) as the number of trials increases. This suggests that "trying harder" at inference time follows the same fundamental mathematical patterns as "learning more" during training.

Inference loss $\mathcal{L}_{inference}$ as a function of the number of trials $k$, demonstrating power-law decay across different LLMs.

The parameters of the Beta distribution ($\alpha$ and $\beta$) act as a "difficulty fingerprint" for a dataset relative to a specific model. By fitting the theoretical pass@k curve to empirical data, researchers can back-calculate the underlying distribution of task difficulty, providing a deeper understanding of model-task alignment.

Empirical Validation

The theoretical model was validated using performance curves from state-of-the-art LLMs, including Llama-3-8B, Gemma-2B, and Pythia-2.8B, on MATH and coding tasks. The results show that the proposed mathematical model fits the empirical data almost perfectly.

Pass@k scaling for various LLMs on the MATH dataset. The dashed lines represent the theoretical fit, showing high agreement with empirical performance.

For challenging mathematical problems, the "hardness" exponent $\beta$ was found to be approximately 0.35 across different architectures, suggesting a universal scaling behavior for these types of reasoning tasks. To further test the theory in a controlled environment, the author conducted experiments with a Variational Autoencoder (VAE) on the Fashion-MNIST dataset. These experiments successfully demonstrated that the eigenvalues of the trial correlation matrix follow a power law, justifying the "effective $k$" approach.

Conclusion and Future Directions

This framework provides a theoretical basis for the "compute trade-off" in AI development. By understanding inference scaling laws, engineers can determine the optimal balance between model size and inference attempts. For example, it may be more cost-effective to use a smaller, cheaper model many times rather than a larger, more expensive model once.

While the current model assumes an "oracle verifier," future work will aim to incorporate the costs and noise associated with real-world verification. Ultimately, this framework sets the stage for a unified theory that integrates inference scaling with established training scaling laws, leading to more efficient and predictable AI systems.

Summary updated

Abstract

Neural scaling laws have garnered significant interest due to their ability to predict model performance as a function of increasing parameters, data, and compute. In this work, we propose a simple statistical ansatz based on memorization to study scaling laws in the context of inference, specifically how performance improves with multiple inference attempts. We explore the coverage, or pass@k metric, which measures the chance of success over repeated attempts and provide a motivation for the observed functional form of the inference scaling behavior of the coverage in large language models (LLMs) on reasoning tasks. We then define an "inference loss", which exhibits a power law decay as the number of trials increases, and connect this result with prompting costs. We further test our construction by conducting experiments on a simple generative model, and find that our predictions are in agreement with the empirical coverage curves in a controlled setting. Our simple framework sets the ground for incorporating inference scaling with other known scaling laws.
BibTeX
@misc{levi2024simple,
  author = {Noam Levi},
  title = {{A Simple Model of Inference Scaling Laws}},
  year = {2024},
  eprint = {2410.16377},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.16377},
}
Figure