Abstract
Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge. Our procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. Despite a modest setup (Qwen-2.5-7B backbone with LoRA rank 8), MachineLearningLM outperforms strong baselines like GPT-5-mini by an average of 15% on out-of-distribution tabular classification. It exhibits a striking many-shot scaling law, where accuracy increases monotonically as demonstrations grow from 8 to 1,024, reaching random-forest-level performance without task-specific training.
Introduction
Standard LLMs are surprisingly poor at traditional machine learning tasks, such as classifying tabular data. They often plateau after just a few examples, are highly sensitive to the order of demonstrations, and struggle with numerical modeling. Specialized models like TabPFN exist to solve this, but they are "black boxes" that lack the world knowledge and natural language interface of an LLM. MachineLearningLM addresses this by teaching an LLM to "do ML in context" through synthetic pretraining and teacher distillation, creating a model that can reason about data while maintaining its chat capabilities.
Methods
The framework transforms a Qwen-2.5-7B-Instruct backbone into a robust tabular predictor using several key techniques:
- Continued Pretraining on SCMs: The model is trained on 3 million synthetic tasks generated from Structural Causal Models (SCMs). This ensures the model learns underlying statistical dependencies rather than just memorizing patterns.
- Random Forest (RF) Distillation: A "warm-up" phase trains the LLM to match the predictions of a Random Forest teacher, distilling robust, tree-based decision strategies into the transformer architecture.
- Token-Efficient Prompting: To maximize context window utility, data is presented in a compact, comma-separated format. Numbers are normalized to integers [0, 999] and mapped to single tokens, preventing the "tokenization fragmentation" that typically degrades LLM numerical performance.
- Sequence-Level Batch Prediction: The model predicts multiple test queries in a single forward pass, increasing throughput by up to 50x and amortizing the cost of long many-shot prompts.
Key Results
MachineLearningLM demonstrates significant improvements over both its base model and much larger proprietary models across the TALENT benchmark (200 real-world tasks).
Performance and Scaling
The model outperformed its base (Qwen-2.5-7B) by an average of ~15% on out-of-distribution tabular tasks. At high shot counts (128–1,024), it surpassed GPT-5-mini by ~12% and o3-mini by ~16%.
A key finding is the monotonic scaling law: unlike vanilla LLMs which often see diminishing returns after 32 shots, MachineLearningLM's accuracy continues to climb as the number of shots increases up to 1,024. Without any task-specific fine-tuning, the model reaches the accuracy levels of a tuned Random Forest.
General Ability Preservation
Crucially, the model retains its general-purpose capabilities. It achieved 75.4% on the MMLU benchmark, proving that specialized ML training does not necessarily lead to catastrophic forgetting of general knowledge and reasoning.
Discussion and Future Work
MachineLearningLM proves that LLMs can be effective "in-context learners" for structured data. The model's ability to generalize from a 32k training budget to 131k tokens at inference suggests it has learned a robust strategy for processing long sequences of statistical data. While currently focused on classification, future expansions will target regression, ranking, and multimodal tables containing images or geospatial signals.
Summary updated