Zero-Shot Weight-Space Programming for Prediction and Classification of Target Variables in Structured Datasets

Eduardo C. Garrido-Merchán

TL;DR

This paper introduces a provocative new capability for Large Language Models (LLMs): Zero-Shot Weight-Space Programming. Instead of using gradient descent, the authors show that an LLM can ingest a small dataset and directly output the literal weights and biases of a functional neural network that generalizes to unseen data, sometimes outperforming standard training methods.

What Problem Does This Paper Address?

In traditional machine learning, we find the parameters of a model through an optimization process—usually gradient descent. Recently, "In-Context Learning" (ICL) has allowed LLMs to make predictions by simply looking at examples in their prompt. However, in ICL, the "model" remains implicit and trapped within the LLM's activations; you can't "export" it.

The authors explore a middle ground called Weight-Space Programming (WSP). Previously, WSP was used to "compile" known rules (like the rules of Chess) into neural network weights. This paper asks a much harder question: Can an LLM perform supervised learning? That is, can it look at raw data, infer the underlying statistical patterns, and then "write out" the weights of a separate, frozen feed-forward network that solves the task—all without a single step of backpropagation or code execution?

Methods

The researchers established a rigorous protocol to test this "reasoning-based" parameter initialization:

  1. The Prompt: An LLM (primarily Claude 3.5 Opus, but also Grok 4.7 and others) is given a small structured dataset (rows of features and targets) and a target architecture (e.g., a linear model or an MLP with 8 hidden units).
  2. The Task: The model must write every weight and bias as a literal number.
  3. The Constraints: No optimizer, no gradient descent, no code execution, and no opportunity for the model to "test" its weights before submitting them.
  4. Scientific Rigor: To prevent "cheating" via trial and error, every construction was "sealed" with a SHA-256 digest and the model had to pre-register its expected accuracy before any evaluation was run.
  5. Baselines: The LLM-generated weights were compared against standard baselines like Ordinary Least Squares (OLS), Logistic Regression, XGBoost, and the same architecture trained via Adam (gradient descent).

Datasets

The study uses a "difficulty-graded ladder" of 24 constructions across twelve rungs:

  • Toy Tier: Simple linear regression and classification problems.
  • Synthetic Tier: Nine problems designed to test specific geometric structures (e.g., Parity/XOR, radial boundaries, spirals, and sinusoids). These were disguised using hidden rotations and scaling to ensure the model couldn't rely on memorized patterns.
  • Benchmark Tier: Classical datasets like Iris, Wine, and Breast Cancer Wisconsin (using anonymized/transformed versions as a control for memorization).
  • Real Tier: A complex engineering regression task involving concrete compressive strength (1,005 rows).

Key Insights and Ideas

  • Inference as Reasoning: The models didn't just guess; they performed explicit statistical reasoning. For example, on the toy_line task, the model sorted the data, calculated group means, and derived the slope and intercept manually.
  • The "Oblique" Wall: Most models failed when the data structure was "oblique" (not aligned with the coordinate axes). If a model analyzed features one-by-one, it missed diagonal relationships.
  • Frame Recovery: Interestingly, the failure on oblique data is a procedural choice, not a fundamental limit. Grok 4.7 succeeded where Claude Opus failed by explicitly "recovering a frame"—essentially performing a mental rotation of the data to find the correct fold directions.
  • Linearity is the Gatekeeper: The authors found that the training error of a linear separator ($L_2$) is the best predictor of whether a hidden layer is actually useful. If $L_2$ is high, the model knows it needs to "spend" its hidden units on non-linearities.

Key Results

  • Matching the Optimum: On linear problems, the LLM-written weights were nearly indistinguishable from the closed-form OLS solution.
  • Beating Gradient Descent: On non-linear synthetic rungs like s3_xor (parity) and s4_ball (radial), the LLM-written networks reached the "irreducible error" ceiling, outperforming the same architecture trained by gradient descent.
  • Real-World Performance: On the concrete regression task, Claude Opus achieved an $R^2$ of 0.7444, coming remarkably close to the gradient-trained MLP (0.7892) and far exceeding OLS (0.5697).
  • Self-Calibration: Stronger models (Opus, Grok) were generally pessimistic about their performance, while weaker models (Haiku) were wildly overconfident, often predicting high accuracy for networks that performed worse than random guessing.

Limitations and Future Work

  • Context Window Limits: This method is currently restricted to "small" data (hundreds to a few thousand rows) that can fit within an LLM's context window.
  • Architectural Complexity: The study focused on very small networks (8 hidden units). It remains to be seen if LLMs can "program" deeper or wider architectures like ResNets or Transformers.
  • The "Oblique" Problem: While Grok showed it's possible, most models still struggle with non-axis-aligned data. Providing the model with a better "vocabulary" for oblique cases is a suggested next step.
  • Scaling Laws: The authors suggest future work should map the "capability curve"—how much reasoning effort and model scale are required to solve increasingly complex datasets via weight-space programming.

Summary updated

Abstract

Weight-space programming fixes every parameter of a feed-forward network by hand, with no gradient descent anywhere in the construction. Every demonstration so far starts from a target function handed to the model as a rule system, making it a compilation problem. A supervised dataset withholds that specification, so the procedure must do the statistical inference itself and then express the result in the parameterisation it was given. We ask whether a large language model can do both. It is shown a small dataset in full, with the architecture of a feed-forward network, and writes out every weight and bias as a literal number, with no optimizer, no gradient, no code execution, and no chance to evaluate it first. Each construction is sealed and pre-registers its expected score. The graded suite runs from two toy problems, through nine synthetic problems disguised by a hidden rotation, to the classical benchmarks and one real engineering regression. The frozen network matches the closed-form optimum wherever that optimum is linear, reaches the irreducible error of the generating process on three rungs where it is not, and beats every fitted baseline on two of those, including the same architecture trained by gradient descent. It fails on two rungs, both times because the structure to be found is oblique to the coordinates it analysed. Three further constructors separate two failure modes a single score conflates: one returns no parameter vector at all on the hardest rungs, another delivers everywhere while writing networks worse than predicting the mean. One of those failures proves avoidable: another constructor solves the oblique-hinge rung by a margin that survives multiple-comparison correction. What predicts whether the hidden layer earns its keep is the training error of a linear separator, which the composite complexity index misses.
BibTeX
@misc{garridomerchan2026zero,
  author = {Eduardo C. Garrido-Merchán},
  title = {{Zero-Shot Weight-Space Programming for Prediction and Classification of Target Variables in Structured Datasets}},
  year = {2026},
  eprint = {2611.19292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19292},
}
Figure