TL;DR
Researchers have demonstrated that frontier LLMs like Claude 5.5 and GPT-6 can "write" the weights of a medical image classifier from scratch using only natural language medical knowledge. While these hand-coded networks outperform massive zero-shot foundation models on several tasks, they still cannot match the performance of a simple network trained on just 256 labeled images.
What Problem Does This Paper Address?
Deep learning in medicine faces a massive bottleneck: data. Labeling medical images requires expensive expert time from radiologists, and for rare diseases, sufficient data may simply not exist.
This paper explores a radical alternative called Weight-Space Programming (WSP). Instead of using an optimizer (like Adam) to find weights by looking at thousands of images, can we ask a Large Language Model (LLM) to directly write the parameters of a neural network based on its internal "textbook" knowledge? For example, if a textbook says a "retroflexed uterus bends backward," can an LLM translate that geometric rule into the specific convolutional filters and weights needed to detect it in an MRI?
Methods
The authors tested two frontier models—Claude Opus 5.5 and GPT-6 Astra—in a highly restrictive setting: a single reply, no tools (no code execution), and zero access to images.
To make this possible, they designed ProgNet, a "programmable" LeNet-style CNN small enough to fit within an LLM's context window (ranging from 591 to 6,974 parameters). The architecture consists of:
- Convolution: 10 filters (3x3) to detect basic features.
- ReLU: Activation.
- Pooling: 4x4 average pooling to reduce the image to a 7x7 grid.
- Read-out: A linear layer where the LLM assigns "evidence" values to specific grid locations for each class.
The LLM was prompted with the dataset description and the network architecture, then asked to provide the weights in a specific text format.
Datasets
The study evaluated performance across 7 classification tasks in four modalities:
- MedMNIST v2: Six 2D radiology tasks including Chest X-rays (ChestMNIST, PneumoniaMNIST), Breast Ultrasound (BreastMNIST), and Abdominal CT scans (OrganA/C/S).
- SynthUterus: A pelvic MRI dataset (anteflexion vs. retroflexion). This was chosen specifically because it is a newer, synthetic dataset with "low exposure" in LLM training data, serving as a test for actual reasoning versus memorization.
Key Insights and Ideas
- "Textbook" vs. "Learned" Weights: LLM-written weights are sparse and discrete. While gradient descent learns smooth, complex, and dense patterns, LLMs tend to use a few rounded values (like ±0.25 or ±0.5) and classic kernels like edge detectors or identity filters.
- Interpretability by Construction: Unlike standard "black box" models, every weight in a WSP network has a stated radiological rationale. You can trace a prediction back to a specific filter the LLM designed to detect "hypoechoic masses" or "opacified lungs."
- The "Guesswork" Gap: A major challenge for LLMs is the lack of spatial context. Without being told the orientation (e.g., "anterior is left"), models can make systematic errors. GPT-6 Astra, for instance, failed the SynthUterus task because it assumed a different anatomical layout than what was present in the 28x28 crops.
- Hidden Knowledge: Interestingly, the LLMs correctly guessed the class distributions (priors) for MedMNIST datasets even though they weren't in the prompt, suggesting the models "remember" these specific benchmarks from their training data.
Key Results
- Functional Classifiers: LLM-written networks performed above chance on all six MedMNIST tasks. Claude Opus achieved a mean AUC of 0.719, while GPT-6 Astra reached 0.711.
- Beating Foundation Models: On 5 out of 6 MedMNIST tasks, these tiny, hand-written networks (under 7k parameters) outperformed MedSigLIP, a 400-million-parameter medical vision-language model used zero-shot.
- Data Still Wins: Despite the impressive zero-shot performance of WSP, the same ProgNet architecture trained with the Adam optimizer on just 256 labeled images achieved a significantly higher mean AUC of 0.828.
Limitations and Future Work
The primary limitation is scale. Currently, LLMs can only write very small networks for low-resolution (28x28) images because every weight must fit in a single text response. As LLM context windows and reasoning capabilities grow, we might see WSP applied to larger, more complex architectures.
The authors suggest that WSP's true value may not be in replacing training data for common tasks, but in providing a transparent, auditable starting point for rare diseases where data is non-existent. A written network could serve as a "day-one" classifier that radiologists can review and refine before the first label is even collected.
Summary updated