Pathology vision-language models learn by matching tissue images to text. Examples are CONCH and MI-Zero. A vision-language model here means an image encoder and a text encoder trained so that matching image-caption pairs land close together in a shared space. These models currently depend on large collections of real slides and clinical reports, which carry patient-privacy and copyright burdens. Even with that data they are fragile: rewording a prompt can swing CONCH's balanced accuracy by up to 26%, and its calibration error stays above 0.5.
A separate line of work, formula-driven supervised learning, pretrains image encoders on pictures generated purely from math, such as fractals and sinusoidal contours, with labels taken from the generating parameters. These synthetic images carry no privacy or licensing risk and transfer well. However, they have never been paired with language. This note proposes the missing step. Every procedural image would be rendered together with a procedurally generated caption that describes its generating parameters in compositional, paraphrased geometric language: "densely packed small irregular blobs with weak orientation" or "branching structure, depth four, smooth boundaries." A vision-language model would be warm-started on this, then aligned to a small, openly licensed pathology corpus.
The bet is that teaching the image side to bind structural attributes to many phrasings of text, before it ever sees tissue, will make the final model less sensitive to prompt wording and better calibrated. A second test is whether this holds across convolutional, transformer, hybrid and MLP backbones rather than only the one it was tuned on.
Core Claim
At a fixed budget of real image-text pairs, contrastive pretraining on formula-generated images with grammar-generated attribute captions should do better than the same images with class labels only, and better than ImageNet initialization. It should give a better starting point for pathology alignment, with the largest gains on prompt-paraphrase robustness and expected calibration error rather than on peak accuracy. The gain should also be roughly backbone-independent.
The specific novelty is the caption channel. It is attached to formula-driven data and evaluated on domain vision-language robustness. No paper in the file pairs formula-driven images with text. Pre-Training Without Natural Images, Visual Atoms and SegRCDB all use parameter-derived class labels for image-only encoders. Formula-Supervised Visual-Geometric Pre-training couples fractal images with 3D point clouds, not language.
The closest multimodal synthetic work is SynCXR. It reports a 3.8% average zero-shot AUC gain over MIMIC-CXR training using diffusion and LLM-generated chest X-ray pairs. But its generators were themselves trained on real clinical and web data, so the provenance problem moves rather than disappears. SynthVLM has the same structure in the general domain. Our generator is closed-form on the image side. The captions are exact by construction, so no filtering cascade is needed to catch hallucinated pairs.
Research Program
The generator. We would extend three existing formula families. Iterated-function-system fractals give branching and self-similar structure. Visual-atom sinusoidal contours give boundary irregularity and frequency. InsCore-style hollow masks give occlusion and packing (Industrial Synthetic Segment Pre-training).
Each family would gain a shared attribute layer. The attributes would be:
- instance count and density
- size variance
- boundary roughness
- branching depth
- orientation coherence
- nested hollow structure
- stain-like color mixing in two channels
These attributes are chosen because they are the geometric analogues of what pathologists describe: cellularity, pleomorphism, glandular branching and architectural disorder. We deliberately use neutral geometric words rather than fake medical terms. The mapping to clinical vocabulary is left to the real alignment stage, and claims about what the warm start teaches stay honest.
The captions. A small context-free grammar would turn the attribute vector into captions. It would vary word order, use synonyms, include or omit attributes, phrase some attributes as negations ("no branching"), and use comparatives. This gives each image many correct paraphrases, and paraphrase invariance becomes something the model is trained on directly.
Training stages. Stage one trains a CLIP-style contrastive objective, plus an image-text matching head as in PathologyVLM, on a few million procedural pairs. Stage two aligns on a fixed, openly licensed pathology image-caption set of a few tens of thousands of pairs, comparable in scale to the ARCH-derived set MI-Zero used. Whole-slide zero-shot inference then uses MI-Zero's top-K pooling, following MI-Zero.
The text encoder is held constant across all arms: the same openly released biomedical language model, so that differences are attributable to the image-side warm start. The vision-language interface is a pooled linear projection rather than a query transformer, so that it applies to every backbone. Backbones are ResNet-50, ViT-B, Swin-T, ConViT and gMLP, matching the families in Evaluation of FractalDB Pre-training with Vision Transformers.
Backbone sensitivity. That paper found that Swin and ConViT fail to learn when fractal complexity is too low, while gMLP does not. We would test whether caption supervision moves that failure threshold. Text that names density and branching gives a gradient signal even when the rendered images are sparse, so the threshold could plausibly shift.
Baselines
Naive reference: ImageNet-1k initialization. The do-nothing reference initializes each backbone from supervised ImageNet-1k weights and runs only stage two on the same real pathology pairs, with the same steps, text encoder, pooling, and hyperparameter search. This is what most practitioners would do. Any proposal that cannot beat it on robustness metrics is not worth the generator.
Key ablation: formula labels only. The most important ablation is the strongest internal competitor. It uses the identical procedural image stream with parameter class labels and a classification loss, exactly as in standard formula-driven pretraining, matched in images seen and compute, then the same stage two. This isolates the caption channel. Beating it on paraphrase robustness is the proposal's main claim.
Shuffled-caption control. Captions are permuted across images, keeping the vocabulary statistics but destroying the binding between images and attributes. If this control matches real captions, the gain comes from text-encoder adaptation rather than attribute grounding.
Real-tissue self-supervised encoder. This uses the setup of an encoder pretrained without labels on real tissue patches, the CTransPath-style recipe behind MI-Zero, followed by the same stage two. It is the strongest competitor that still avoids paired text, though it uses real patient images. It tells us what the license-free route costs.
Published reference points, not matched-budget competitors. CONCH reports 84.0% zero-shot accuracy on TCGA BRCA, trained on 1.17 million real pairs (CONCH). MI-Zero reports a 70.2% average median zero-shot accuracy across BRCA, NSCLC and RCC subtyping (MI-Zero). For robustness, Histo-VL reports up to 26% balanced-accuracy fluctuation under caption variation, up to 20% accuracy drops at adversarial ε=0.3, and ECE of 0.53–0.61 for CONCH.
We do not expect to beat CONCH's accuracy with an order of magnitude less real data. The success criteria are three:
- clear gains over ImageNet initialization and over the labels-only ablation in paraphrase spread and ECE, at matched data, on every backbone;
- zero-shot TCGA accuracy within reach of MI-Zero's 70.2%;
- paraphrase spread well under the 26% Histo-VL reports for CONCH.
Why not just…?
Why not just pretrain the encoder on formula images with labels, then align on real pairs? This is the real threat. It may capture all of the shape bias that matters, as the radiolarian study suggests by reporting 86.30% for radial-contour pretraining versus 85.61% for ImageNet. The labels-only and shuffled-caption arms settle it directly. If captions give no robustness or calibration gain over labels on most backbones, the claim fails. In that case we would report that the language channel is unnecessary for formula-driven warm starts.
Why not just synthesize realistic tissue with diffusion, as SynCXR did for chest X-rays? SynCXR shows this can work for accuracy. However, it relies on RoentGen and Llama 3.1, both trained on uncurated real data, and it needs an automated filter to remove hallucinated pairs. The question here is how far a provenance-clean generator gets. We report the gap to the real-tissue self-supervised encoder honestly rather than claiming parity with diffusion. No pathology diffusion result appears in the file, so we cannot claim to beat it.
Why not just fix brittleness at test time with prompt ensembling or temperature scaling? Histo-VL reports that CONCH's ECE rose from 0.53 to 0.61 even as ensembling improved accuracy. That is evidence that prompt ensembling does not repair calibration. Post-hoc temperature scaling needs labeled target data, which zero-shot use lacks. We will nonetheless apply both to every arm, and claim a gain only if it survives them.
Why not just fine-tune CLIP with LoRA on the pathology pairs? CONCH reports that general models sit near 50–55% on BRCA subtyping. We include LoRA-tuned CLIP at the same real-data budget as an extra reference. It cannot answer the backbone question, because CLIP fixes the architecture.
Proposed Experiments
Primary evaluation. For each of the five backbones and each arm we measure:
- zero-shot TCGA BRCA, NSCLC and RCC whole-slide accuracy with MI-Zero pooling;
- on the Histo-VL tasks, the spread in balanced accuracy across a fixed set of 20 clinician-style prompt paraphrases per class;
- ECE;
- accuracy under ε=0.3 image and text attacks, following the Histo-VL protocol.
Results are reported over three seeds.
Decisive check one: the caption channel. If captions beat labels-only on paraphrase spread and ECE on at least four of five backbones, while shuffled captions do not, attribute grounding is supported.
Decisive check two: architecture. Following Nakashima et al., we sweep fractal rendering density downward. If caption supervision lowers the density at which Swin and ConViT stop learning, that is a mechanistic finding about backbones, whatever the downstream numbers.
Secondary probe. We test whether gains track attribute relevance. Dropping the density and size-variance attributes from the grammar should hurt grading and subtyping more than dropping orientation does. If no such selectivity appears, the "geometric analogue" story is wrong even if aggregate numbers improve.
References
- Pre-training Without Natural Images
- Visual Atoms: Pre-Training Vision Transformers with Sinusoidal Waves
- SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learning
- Formula-Supervised Visual-Geometric Pre-training
- Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?
- SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models
- Industrial Synthetic Segment Pre-training
- Pathologyvlm: a large vision-language model for pathology image understanding
- Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
- Evaluation of FractalDB Pre-training with Vision Transformers
- A visual-language foundation model for computational pathology
- How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark
- Classifying microfossil radiolarians on fractal pre-trained vision transformers