MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining

Haoyu Dong, Pengkun Zhang, Mingzhe Lu, Yanzhen Shen, Guolin Ke

Abstract

Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge. Our procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. Despite a modest setup (Qwen-2.5-7B backbone with LoRA rank 8), MachineLearningLM outperforms strong baselines like GPT-5-mini by an average of 15% on out-of-distribution tabular classification. It exhibits a striking many-shot scaling law, where accuracy increases monotonically as demonstrations grow from 8 to 1,024, reaching random-forest-level performance without task-specific training.

Introduction

Standard LLMs are surprisingly poor at traditional machine learning tasks, such as classifying tabular data. They often plateau after just a few examples, are highly sensitive to the order of demonstrations, and struggle with numerical modeling. Specialized models like TabPFN exist to solve this, but they are "black boxes" that lack the world knowledge and natural language interface of an LLM. MachineLearningLM addresses this by teaching an LLM to "do ML in context" through synthetic pretraining and teacher distillation, creating a model that can reason about data while maintaining its chat capabilities.

Methods

The framework transforms a Qwen-2.5-7B-Instruct backbone into a robust tabular predictor using several key techniques:

  1. Continued Pretraining on SCMs: The model is trained on 3 million synthetic tasks generated from Structural Causal Models (SCMs). This ensures the model learns underlying statistical dependencies rather than just memorizing patterns.
  2. Random Forest (RF) Distillation: A "warm-up" phase trains the LLM to match the predictions of a Random Forest teacher, distilling robust, tree-based decision strategies into the transformer architecture.
  3. Token-Efficient Prompting: To maximize context window utility, data is presented in a compact, comma-separated format. Numbers are normalized to integers [0, 999] and mapped to single tokens, preventing the "tokenization fragmentation" that typically degrades LLM numerical performance.
The MachineLearningLM prompt format uses token-efficient encoding and integer normalization to fit up to 1,024 shots in context.
  1. Sequence-Level Batch Prediction: The model predicts multiple test queries in a single forward pass, increasing throughput by up to 50x and amortizing the cost of long many-shot prompts.

Key Results

MachineLearningLM demonstrates significant improvements over both its base model and much larger proprietary models across the TALENT benchmark (200 real-world tasks).

Performance and Scaling

The model outperformed its base (Qwen-2.5-7B) by an average of ~15% on out-of-distribution tabular tasks. At high shot counts (128–1,024), it surpassed GPT-5-mini by ~12% and o3-mini by ~16%.

MachineLearningLM shows significant accuracy gains across diverse tabular tasks (top) and exhibits a monotonic scaling law as the number of in-context shots increases (bottom).

A key finding is the monotonic scaling law: unlike vanilla LLMs which often see diminishing returns after 32 shots, MachineLearningLM's accuracy continues to climb as the number of shots increases up to 1,024. Without any task-specific fine-tuning, the model reaches the accuracy levels of a tuned Random Forest.

General Ability Preservation

Crucially, the model retains its general-purpose capabilities. It achieved 75.4% on the MMLU benchmark, proving that specialized ML training does not necessarily lead to catastrophic forgetting of general knowledge and reasoning.

Discussion and Future Work

MachineLearningLM proves that LLMs can be effective "in-context learners" for structured data. The model's ability to generalize from a 32k training budget to 131k tokens at inference suggests it has learned a robust strategy for processing long sequences of statistical data. While currently focused on classification, future expansions will target regression, ranking, and multimodal tables containing images or geospatial signals.

Summary updated

Abstract

Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.
BibTeX
@misc{dong2025machinelearninglm,
  author = {Haoyu Dong and Pengkun Zhang and Mingzhe Lu and Yanzhen Shen and Guolin Ke},
  title = {{MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining}},
  year = {2025},
  eprint = {2509.06806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.06806},
}
Figure