Arrows as Labels: Harvesting Grounding Supervision from Published Case Reports

When a clinician publishes a case report about a rare disease, the figures usually already carry an expert annotation. An arrow points at the pontine enhancement, an asterisk marks the opacified sphenoid sinus, and the caption names the finding: "arrows indicate perivascular enhancement." A subspecialist drew that mark during peer review at no cost to any machine learning project, and millions of such marks sit in the open-access literature. Many case reports also add a literature review that collects 15 to 60 previously published patients with the same condition and summarizes them as cohort statistics. One example is the sinonasal extramedullary hematopoiesis review reporting maxillary involvement in 60% of 15 patients. This proposal treats each arrow–caption pair as a free point label for visual grounding, meaning a model that, given a phrase, outputs the box in the image where that finding is. It treats the review statistics as a group-level check on those labels. The aim is to train grounding models for rare findings where no expert-boxed training set exists or ever will.

The argument for doing this is about scale across diseases rather than scale within one disease. For any single rare condition, an expert could box its few dozen published figures in an afternoon. There are thousands of such conditions, though, and each needs a different subspecialist. The literature has already paid that cost once, in a form that resembles a familiar trick. Radiologists' routine RECIST diameter lines were repurposed as weak segmentation labels in Weakly Supervised Lesion Co-Segmentation on Ct Scans. Published arrows are the literature's equivalent of RECIST lines.

Core Claim

Arrow tips detected in case-report figures, paired with the caption phrase they illustrate, can replace the manually boxed "cold start" that reinforcement-learned grounding models currently require. They can also extend grounding to rare findings where no boxes exist.

This holds only under one condition: the arrows are removed from the image before training. Otherwise a small-sample model will learn to find arrows instead of pathology. This shortcut is exactly the failure mode documented for chest X-ray models (Unveiling COVID-19 from Chest X-ray with deep learning: a hurdles race with small data Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data) and targeted by SpurCon. A secondary claim is that the cohort proportions from literature reviews add a useful constraint when a disease has fewer than about 50 harvested figures. This secondary claim is weaker and is tested as an ablation.

Relation to Prior Work

Two lines of work motivate the gap. The first is text-only case-report systems. FindZebra segments PubMed case abstracts into clinical facets and retrieves them for rare-disease queries. Clinical Case Reports for NLP builds a 717-report French corpus with manual entity labels. RareBench evaluates language models on rare-disease diagnosis with knowledge-graph few-shot prompting. All three discard the figures, and RareBench names multimodal input as its main limitation. Our proposal uses the figures and does not attempt diagnosis from text.

The second line is label-efficient medical vision, which supplies the methods we reuse. MedGround-R1 trains a grounding model with Group Relative Policy Optimization (GRPO), a reinforcement-learning method that scores a group of sampled answers against each other. Its reward combines a box-overlap term with a semantic term from a frozen MedCLIP encoder. It removes the need for expert step-by-step rationales but still needs a supervised cold start on manual boxes, and it handles only short phrases on common chest findings. We keep its training recipe and replace the manual cold start with harvested arrows, plus a point-based reward.

Agarwal et al. ( Weakly Supervised Lesion Co-Segmentation on Ct Scans) seed GrabCut, a classical foreground-extraction algorithm, with RECIST lines to make pseudo-masks. We seed the same algorithm with arrow tips to estimate box extent. Their finding that seed errors propagate during self-training constrains our design, and it is why we do not iterate pseudo-label refinement. Learning to Segment Medical Images with Scribble-Supervision Alone shows that sparse marks can come within 3 to 5 Dice points of full masks on ACDC and NCI-ISBI. This is evidence that point-like supervision can carry enough signal, though their scribbles were drawn deliberately rather than harvested.

The group-level constraint borrows its logic from Yamaguchi et al.. Their Max-Severity Triplet Loss forbids image predictions that contradict a patient-level record. We forbid cohort-level predictions that contradict a published review table.

Prior work does not supply some components. To our knowledge from this file, nothing detects and inpaints arrows, uses arrow tips as grounding labels, or treats literature-review mini-cohorts such as the 60 CLIPPERS patients in Hou et al. or the 15 EMH patients in Clark et al. as supervision. Figure–caption pairs from open literature are used elsewhere for general image–text pretraining. That use gives image-level signal only. The arrow is what turns a caption into a location.

Research Program

The first component is a harvesting pipeline over open-access case reports with figures. A small detector finds arrows, arrowheads, and asterisks and estimates each tip and pointing direction. It is trained on a few hundred hand-marked figures, which is a non-expert task because anyone can see an arrow. Caption parsing then links each mark to a finding phrase, using caption cues such as "arrow," "arrowheads," and panel letters. Multi-panel figures are split, and each panel is assigned an imaging modality. Each mark is removed by inpainting so that the training image no longer contains the cue.

The second component estimates box extent. A box is derived from GrabCut seeded at the tip in the inpainted image. This estimate is used only as a soft prior on box size, never as a hard label, because extent is the least reliable part of the signal.

The third component is training. A cold-start stage fits a vision-language grounding model on (inpainted image, phrase, estimated box) triples. GRPO then follows the MedGround-R1 recipe, except that the binary overlap reward is replaced by a point reward. The point reward equals 1 if the arrow tip falls inside the predicted box, minus a penalty proportional to $|\log(A/\hat{A})|$, where $A$ is the predicted box area and $\hat{A}$ is the GrabCut extent estimate. The MedCLIP semantic term is kept to reward near-misses.

The fourth component is the cohort constraint, used for diseases with an accompanying review. Let $\pi_k$ be the reported fraction of review patients with involvement of anatomical region $k$, for example the maxillary sinus. Let $q_k$ be the fraction of that disease's harvested figures in which the model's predicted box falls in region $k$, with region assignment from an atlas or a coarse region classifier. We add

$$\mathcal{L}_{\text{cohort}} = \sum_k \pi_k \log \frac{\pi_k}{q_k},$$

which is a KL divergence pulling the model's location distribution toward the published one. With only 15 to 60 patients behind each $\pi_k$, this term is weighted by review size and is expected to matter only at the smallest scales.

Two known confounders shape the design. Case reports select dramatic presentations, so all evaluation uses images that are not from case reports wherever possible. Older reports can describe outdated disease courses, as FindZebra found for Gaucher disease. We therefore record publication year and report results stratified by era, without hard-filtering by date.

Baselines

The naive reference point is the same vision-language backbone used zero-shot, together with a phrase-blind prior that predicts the training-set mean box for each anatomical region. Any method that fails to clearly beat these on rare findings has learned nothing useful from the harvest.

The strongest existing method is MedGround-R1 trained with its usual manual-box cold start. On common chest findings it reports 79.02 mIoU on MS-CXR, 53.12 on ChestX-ray8, and 60.10 on M3D-RefSeg (2D adapted), compared with BiRD at 73.33, 48.22, and 52.02. We rerun it with the same backbone, the same number of GRPO steps, and the same hyperparameter search budget as our variant. The only difference is the cold-start data and the reward.

On common chest benchmarks, "beating" it is not the goal. Matching it within roughly 3 mIoU on MS-CXR without manual boxes would show that harvested arrows substitute for expert boxes. On rare findings, where no manual cold start exists, the comparison is MedGround-R1 cold-started on common-finding boxes and then applied to rare phrases. Beating that comparison by a margin larger than the seed-to-seed spread across five seeds is the target.

A third baseline is Agarwal-style pseudo-label training: GrabCut boxes from arrow tips used directly as supervised targets, with no reinforcement learning and no point reward. It isolates whether the point reward helps beyond the pseudo-boxes themselves.

Three ablations isolate the key ingredients. The first trains on images with the arrows left in. This isolates inpainting and is the shortcut test. The second drops the cohort term. The third is caption-only training, in which the arrow is discarded and only image-level phrase matching is used. This isolates the value of the location signal.

Why not just…?

Why not just have an expert box the published figures directly? For a single disease this is cheaper and better, and we expect it to win at equal image counts. The proposal's value claim is about expert-minutes across hundreds of findings. We will time expert boxing on a sample and report a crossover curve: how many harvested arrow labels match the accuracy of $N$ expert boxes, for $N$ from 10 to 200. If 10 expert boxes per disease beat the full harvest, the harvesting pipeline is not worth building, and we will say so.

Why not just pretrain contrastively on figure–caption pairs and ground from attention maps? Captions say what is in a figure but not where it is. The caption-only ablation tests this directly. Its failure would mirror the general result in Yamaguchi et al., where adversarial alignment without structured clinical constraints reached 52.1% against 71.4% with them. That result is evidence about structured constraints rather than proof for grounding, which is why the ablation is run.

Why not just diagnose from the text, as RareBench and FindZebra do? Text-based diagnosis answers a different question. Grounding is needed when a radiologist has to find the rare finding in a new scan. RareBench explicitly lists missing imaging as its main limitation. This is not a competing baseline for our metric, but we report it so readers can see the two outputs are complementary.

Why not just reuse routine hospital annotations like DeepLesion's RECIST lines? Those cover common lesions from one institution. As DeepLesion notes, they also mark only the primary finding. Rare-disease findings are essentially absent from them. We include a DeepLesion-pretrained detector as an extra comparison on rare findings, to measure how much generic lesion knowledge already transfers.

Proposed Experiments

The first experiment tests substitution on common findings. We harvest arrowed chest radiograph figures from case reports, train with no manual boxes, and evaluate on MS-CXR and ChestX-ray8 against the reported 79.02 and 53.12 mIoU and against our controlled rerun of MedGround-R1.

The second experiment targets rare findings, for which there is no reported result on Lacuna. We build a test set of roughly 30 rare findings drawn from conditions such as CLIPPERS and sinonasal extramedullary hematopoiesis. Each finding has 20 to 40 test images boxed by subspecialists. These images come from figures with no arrows and, where obtainable, from clinical scans. We exclude any figure whose source report appears in training. Metrics are mIoU and pointing accuracy, meaning the fraction of predicted boxes that contain the expert-marked center. Results are stratified by the number of harvested training figures per finding, to locate where the cohort term starts to matter.

The decisive check is the shortcut probe. We paint synthetic arrows at wrong locations on test images and measure how often predictions follow the fake arrow rather than the pathology. A model trained without inpainting should fail this probe badly. Ours must not. If ours also follows the arrows, the central claim fails regardless of benchmark mIoU.

A final check stratifies rare-finding accuracy by the publication era of the training sources. This tests whether the temporal decay seen in FindZebra's Gaucher results also appears in imaging supervision.