Weight-Space Programming for Medical Imaging: Do We Still Need Training Data?

Maximilian Lindholz, Richard Ruppel, Annette Milnik, Sebastian Halskov, Semil Eminovic, Charlie A. Hamm, Robin Schmidt, Anika Knupfer, Jana Hutter, Johanna P. Müller, Carolina Dominguez Aleixo, Jawed Nawabi, Jens Vogel-Claussen, Tobias PenzkoferarXiv 2026 · arXiv

TL;DR

Researchers have demonstrated that frontier LLMs like Claude 5.5 and GPT-6 can "write" the weights of a medical image classifier from scratch using only natural language medical knowledge. While these hand-coded networks outperform massive zero-shot foundation models on several tasks, they still cannot match the performance of a simple network trained on just 256 labeled images.

What Problem Does This Paper Address?

Deep learning in medicine faces a massive bottleneck: data. Labeling medical images requires expensive expert time from radiologists, and for rare diseases, sufficient data may simply not exist.

This paper explores a radical alternative called Weight-Space Programming (WSP). Instead of using an optimizer (like Adam) to find weights by looking at thousands of images, can we ask a Large Language Model (LLM) to directly write the parameters of a neural network based on its internal "textbook" knowledge? For example, if a textbook says a "retroflexed uterus bends backward," can an LLM translate that geometric rule into the specific convolutional filters and weights needed to detect it in an MRI?

Methods

The authors tested two frontier models—Claude Opus 5.5 and GPT-6 Astra—in a highly restrictive setting: a single reply, no tools (no code execution), and zero access to images.

To make this possible, they designed ProgNet, a "programmable" LeNet-style CNN small enough to fit within an LLM's context window (ranging from 591 to 6,974 parameters). The architecture consists of:

  1. Convolution: 10 filters (3x3) to detect basic features.
  2. ReLU: Activation.
  3. Pooling: 4x4 average pooling to reduce the image to a 7x7 grid.
  4. Read-out: A linear layer where the LLM assigns "evidence" values to specific grid locations for each class.

The LLM was prompted with the dataset description and the network architecture, then asked to provide the weights in a specific text format.

Datasets

The study evaluated performance across 7 classification tasks in four modalities:

  • MedMNIST v2: Six 2D radiology tasks including Chest X-rays (ChestMNIST, PneumoniaMNIST), Breast Ultrasound (BreastMNIST), and Abdominal CT scans (OrganA/C/S).
  • SynthUterus: A pelvic MRI dataset (anteflexion vs. retroflexion). This was chosen specifically because it is a newer, synthetic dataset with "low exposure" in LLM training data, serving as a test for actual reasoning versus memorization.
Figure 1. The 7 medical imaging tasks (Sec. 3.1), two examples each. For display only, images are shown at higher resolution (MedM- NIST+ [48] at 64 or 224 pixels, SynthUterus at its released 96); the networks see the same images at 28×28.

Key Insights and Ideas

  • "Textbook" vs. "Learned" Weights: LLM-written weights are sparse and discrete. While gradient descent learns smooth, complex, and dense patterns, LLMs tend to use a few rounded values (like ±0.25 or ±0.5) and classic kernels like edge detectors or identity filters.
  • Interpretability by Construction: Unlike standard "black box" models, every weight in a WSP network has a stated radiological rationale. You can trace a prediction back to a specific filter the LLM designed to detect "hypoechoic masses" or "opacified lungs."
  • The "Guesswork" Gap: A major challenge for LLMs is the lack of spatial context. Without being told the orientation (e.g., "anterior is left"), models can make systematic errors. GPT-6 Astra, for instance, failed the SynthUterus task because it assumed a different anatomical layout than what was present in the 28x28 crops.
  • Hidden Knowledge: Interestingly, the LLMs correctly guessed the class distributions (priors) for MedMNIST datasets even though they weren't in the prompt, suggesting the models "remember" these specific benchmarks from their training data.

Key Results

  • Functional Classifiers: LLM-written networks performed above chance on all six MedMNIST tasks. Claude Opus achieved a mean AUC of 0.719, while GPT-6 Astra reached 0.711.
  • Beating Foundation Models: On 5 out of 6 MedMNIST tasks, these tiny, hand-written networks (under 7k parameters) outperformed MedSigLIP, a 400-million-parameter medical vision-language model used zero-shot.
  • Data Still Wins: Despite the impressive zero-shot performance of WSP, the same ProgNet architecture trained with the Adam optimizer on just 256 labeled images achieved a significantly higher mean AUC of 0.828.
Comparison of where LLM-written and Adam-trained networks "look" for evidence. Red indicates evidence for the class.

Limitations and Future Work

The primary limitation is scale. Currently, LLMs can only write very small networks for low-resolution (28x28) images because every weight must fit in a single text response. As LLM context windows and reasoning capabilities grow, we might see WSP applied to larger, more complex architectures.

The authors suggest that WSP's true value may not be in replacing training data for common tasks, but in providing a transparent, auditable starting point for rare diseases where data is non-existent. A written network could serve as a "day-one" classifier that radiologists can review and refine before the first label is even collected.

Summary updated

Abstract

Can a large language model (LLM) write the parameters of a medical image classifier from medical knowledge alone, without any training data? Such classifiers normally require labelled images, which are costly and often unavailable. We investigate this question through weight-space programming (WSP), in which an LLM writes or adapts the weights and biases of a neural network directly, without optimisation on training data. Claude Opus 5.5 and GPT-6 Astra each generate the parameters of a small LeNet-style CNN in a single reply, without tool use and without access to any image. We compare these networks against zero-shot MedSigLIP and against the same architecture trained on labels using the Adam optimiser, across 7 classification tasks in four modalities: chest X-ray, breast ultrasound and abdominal CT (all from MedMNIST), and pelvic MRI (SynthUterus). The LLM-written networks perform above chance on all six MedMNIST tasks, with a mean test AUC of 0.719 for Opus and 0.711 for Astra. Both models outperform MedSigLIP on 5 of 6 tasks while using more than 10,000 times fewer parameters. However, the same network trained with Adam on only 256 labelled images achieves a higher AUC on every MedMNIST task (mean AUC 0.828). On SynthUterus, Opus attains an AUC of 0.764, whereas Astra falls below chance (AUC 0.136) owing to a systematic error. These results show that LLMs can write functional medical classifiers, but that they do not yet replace training data. We release the protocol, logs and all generated weights at https://github.com/MaximilianLindholz/weight-space-programming-medical.
BibTeX
@misc{lindholz2026weight,
  author = {Maximilian Lindholz and Richard Ruppel and Annette Milnik and Sebastian Halskov and Semil Eminovic and Charlie A. Hamm and Robin Schmidt and Anika Knupfer and Jana Hutter and Johanna P. Müller and Carolina Dominguez Aleixo and Jawed Nawabi and Jens Vogel-Claussen and Tobias Penzkofer},
  title = {{Weight-Space Programming for Medical Imaging: Do We Still Need Training Data?}},
  year = {2026},
  eprint = {2611.12066},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12066},
}
Figure