Vision-language models find things. They are good at linking a phrase like "necrosis" or "tumor cells" to a region of an image, and weak at linking a phrase like "immune exclusion", "tumor grade" or "poor prognosis" to anything at all. The reason is that the second kind of concept has no single location. It is a pattern: a particular mix of tissue types arranged in a particular way across the image. The problem is sharpest in digital pathology. A whole-slide image (WSI) is a scan of a tissue section roughly 100,000 pixels on a side, far too large for GPU memory. It is therefore cut into tens of thousands of small tiles and processed as an unordered "bag" of tile features. Current methods that pair slides with language treat grounding as finding the right tiles. That suits focal findings and leaves the clinically most important concepts, which are spatially diffuse, poorly represented.
We propose to ground each abstract pathology concept in the slide-wide spatial co-occurrence pattern of simpler, text-anchored concrete concepts, instead of in a subset of tiles. Each tile is first scored against a small vocabulary of concrete concepts (lymphocyte, tumor epithelium, stroma, fat, necrosis). This uses a frozen pathology vision-language encoder. The scores are then summarized as a concept-by-concept co-occurrence matrix across neighboring tiles. An abstract concept becomes a small learned function of that matrix. The central claim is testable and could fail: the advantage of this "attributional" grounding over tile-selecting attention should grow with how spatially diffuse the target concept is. It should vanish, or reverse, for focal targets such as lymph-node metastasis.
Where this idea comes from and what it is not
The design borrows a finding from lexical semantics. Iosif et al. found that concrete nouns are best compared with a maximum-similarity operator that finds the single best neighbor. Abstract nouns are better compared by correlating their whole profile across a compact set of about ten attribute neighbors. Their result motivates the proposal: it suggests the aggregation operator itself should depend on concreteness. In slide analysis, top-k and quadtree zooming, as in HAG-MIL, act as the maximum-similarity operator. No pathology language model we found switches to a profile-style operator for abstract terms. A second idea is taken directly as method. RDAN showed that relational concepts in natural images (verbs, adjectives) can be matched by treating the region-by-word similarity matrix as a feature map and convolving over it. RDAN only worked with 36 detector regions. Our co-occurrence matrix has a fixed size (vocabulary by vocabulary) regardless of slide size, which is what lets the same intuition run on gigapixel bags.
The closest pathology work constrains the design and shows where the gap is. CATE aligns tile features to text "concept anchors" from the CONCH vision-language model and uses them to filter tiles before standard pooling. It reports roughly 4 to 5% out-of-domain AUC gains on TCGA-BRCA, and its authors state that survival prediction remains open. It uses concepts as per-tile filters. We use them as the alphabet for a spatial pattern. PathFLIP grounds decomposed report subcaptions to local region clusters. Report generators such as HistoGPT and MI-Gen compress the slide into a few tokens and generate text. That compression discards the spatial layout our concepts depend on, and it offers no check that a generated term is supported by the tissue. mSTAR aligns whole slides with reports and gene expression, but only at the level of the whole bag. ConCor-1 grounds text spans and image masks in both directions. It is built for instance masks on natural images, which is exactly the wrong output format for a pattern with no boundary. Tile-correlation models such as TransMIL and the graph model of Adnan et al. do capture spatial relations, but only visually, with no link to language.
The symbol-grounding literature explains why the linguistic anchor matters and why it is not sufficient alone. Merrill and Goldberg prove that the meaning of context-dependent expressions cannot be recovered from linguistic assertions without access to the hidden state they refer to. "Pleomorphic spindle cells" means different things in a carcinoma, in reactive stroma and in a sarcoma. A fixed text anchor applied tile by tile, as CATE does, inherits exactly this ambiguity. Our operator resolves a tile-level term by the company it keeps on the slide, which is a concrete, testable form of the "external state" their theorem requires.
Core Claim
We make three claims. The first is that a slide-level representation built from text-anchored concrete concepts and their spatial co-occurrences grounds diffuse clinical concepts (grade, immune pattern, mutation status, survival risk) better than tile-selecting attention pooling. The second is that it does so while remaining inspectable in the concrete vocabulary: a prediction of "immune exclusion" can be read off as "lymphocyte score high at the tumor–stroma border, low inside tumor". The third is that this advantage scales with the diffuseness of the target and is absent for focal targets. We treat the third claim as the real scientific content. If the gains are uniform across focal and diffuse tasks, we have just built a better pooling layer, not evidence about how abstract concepts should be grounded.
Research Program
Concrete concept scores. Tiles are embedded with a frozen pathology vision-language encoder; CONCH is used, following CATE. Each tile gets a similarity to each of $K \approx 20$ to 30 concrete concept prompts, written by a pathologist and restricted to things visible in a single tile. Per slide this produces $K$ score maps on the tile grid.
Attributional signature. Let $p_i \in \Delta^K$ be the softmax-normalized concept scores of tile $i$, and let $\mathcal{N}_r(i)$ be the tiles at grid distance $r$ from $i$. The signature of the slide at distance $r$ is
$$C^{(r)} = \frac{1}{Z_r}\sum_i \sum_{j \in \mathcal{N}_r(i)} p_i\, p_j^{\top},$$
a $K \times K$ matrix, where $Z_r$ counts the tile pairs. We use a few distances $r \in \{0, 1, 3, 8\}$. At $r=0$ this reduces to composition (how much of each tissue there is). Larger $r$ captures arrangement, such as tumor bordered by lymphocytes versus lymphocytes infiltrating tumor. The cost is $O(NK^2)$ over $N$ tiles with a sparse neighbor lookup, so the whole slide fits comfortably in memory without pruning. This sidesteps the risk, noted for HAG-MIL, that early top-$k$ selection throws away diffuse evidence.
Abstract concept heads. Each abstract concept or clinical target is a small network, a two-layer convolution over the stacked $C^{(r)}$ in the style of RDAN, trained with slide-level labels. For zero-shot use, a language-model embedding of the abstract term's textual definition is mapped to head weights. This is trained on some abstract concepts and tested on held-out ones.
Concreteness index. To test the scaling claim, each target task gets a diffuseness score that is independent of our method. We train an attention-pooling baseline on the target and take the normalized entropy of its attention over tiles, averaged over correctly classified test slides. Low entropy means the evidence sits in a few tiles (focal); high entropy means it is spread out (diffuse).
Baselines
The naive reference point is mean pooling of the same frozen CONCH tile features, with a linear classifier. It shows what the encoder gives for free, and the file's discussion of pooling predicts it will fail on focal targets such as small metastases. We also run a composition-only ablation: our signature restricted to $r=0$, which is simply how much of each tissue type the slide contains.
The strongest tile-selecting competitors are TransMIL and HAG-MIL, both with public code. TransMIL reports a test AUC of 93.09% on Camelyon16, 96.03% on TCGA-NSCLC and 98.82% on TCGA-RCC. HAG-MIL reports AUC 0.946 and F1 0.874 on Camelyon16. These are the strongest because they model tile correlations and multiple resolutions, so any spatial advantage we claim must survive against them. CATE layered on TransMIL is the strongest concept-based competitor, since it uses the same CONCH anchors. Its reported gain is approximately 4 to 5% out-of-domain AUC on TCGA-BRCA.
For fairness, every method uses the same frozen CONCH tile features, the same TCGA patient-level splits and the same hospital-held-out out-of-domain splits, with an equal hyperparameter search budget of 30 configurations each on validation data. Results are averaged over five seeds. Published numbers are quoted only as sanity checks that our reimplementations are sound, because they come from different feature extractors.
Beating the baselines means two things. On diffuse targets, our method must be statistically better than the best of TransMIL, HAG-MIL and CATE+TransMIL, in AUC or concordance index, across seeds and on out-of-domain sites. On Camelyon16, it must be no better than, and possibly worse than, those baselines. For survival, mutation and grade on these cohorts, we found no reported result on Lacuna for TransMIL, HAG-MIL or CATE. mSTAR reports only a relative gain of +2.6% Macro-AUC for mutation prediction over the next-best foundation model. We will therefore establish these baseline numbers ourselves.
The key ablation replaces the text-anchored concept scores with $K$ unsupervised k-means clusters of the same tile features, keeping the co-occurrence signature and heads identical. This isolates whether language anchoring contributes anything beyond spatial statistics.
Why not just…?
Why not just use attention pooling, which can already interpolate between max and mean? Attention can spread weight over many tiles, but it still scores each tile individually. It cannot represent "A next to B", only "some A and some B". TransMIL's position encoding partly addresses this, which is why it is a main baseline. If TransMIL matches us on the high-entropy tasks, the claim that diffuse concepts need a pattern representation is wrong for these tasks, and we will report it that way.
Why not just cluster tiles without language and compute spatial statistics? This is the most serious threat. The k-means ablation tests it directly. If clusters match text anchors in accuracy, the language contributes only interpretability and zero-shot transfer to held-out concepts. We would then narrow the claim to those two properties and evaluate them separately, using the held-out-concept zero-shot test and pathologist ratings of the explanations.
Why not just project a pooled slide embedding into a language model, as a linear map does for natural images? LiMBeR shows such maps work mainly when the vision encoder was trained with language supervision, and that the language model then tends to invent content. More fundamentally, pooling to one vector before projection discards the arrangement our claim depends on. We include a HistoGPT-style compressed-token baseline on the report-derived concept tasks to check this empirically.
Proposed Experiments
The tasks span the focal-to-diffuse axis. At the focal end is Camelyon16 metastasis detection. In the middle is subtyping on TCGA-NSCLC, RCC, BRCA and PRAD. At the diffuse end are histologic grade and survival from TCGA clinical metadata, plus mutation and biomarker status on CPTAC. For a report-grounded evaluation, abstract terms are extracted from PathText reports. Each slide is labeled for the presence of terms such as "desmoplastic" or "infiltrative growth", and 30% of the terms are held out for zero-shot testing.
The decisive check is a regression of our method's gain over the best baseline against the attention-entropy diffuseness index, across all tasks. A significant positive slope, with gains near zero or negative on Camelyon16, supports the claim. A flat slope refutes it.
Two causal checks follow. First, randomly permuting tile positions within each slide should collapse the gains that depend on $r>0$ for relational concepts such as immune exclusion, while leaving composition-driven targets intact. Second, to test context dependence, we select slides where the tile-level concept "spindle cells" is strongly present. We then ask whether its contribution to the malignancy prediction changes sign with the slide's co-occurrence context, as it should under our account and cannot under a fixed tile-level anchor.
Finally, pathologists rate whether the concrete-concept explanation that the head attributes to each prediction is histologically plausible. The comparison is against attention heatmaps from TransMIL on the same slides.
References
- Semantic Similarity Computation for Abstract and Concrete Nouns Using Network-based Distributional Semantic Models
- Diagnose Like a Pathologist: Transformer-Enabled Hierarchical Attention-Guided Multiple Instance Learning for Whole Slide Image Classification
- Multi-Level Visual-Semantic Alignments with Relation-Wise Dual Attention Network for Image and Text Matching
- Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
- Generating clinical-grade pathology reports from gigapixel whole slide images with HistoGPT
- WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images
- A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model
- Vision-Language Grounding as Bidirectional Concept Correspondence
- TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification
- Representation Learning of Histopathology Images using Graph Neural Networks
- Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?
- Linearly Mapping from Image to Text Space