Weak Lesion Segmentation as Node Classification on a Typed Graph

Radiologists do not outline tumors pixel by pixel. They draw two lines: the lesion's longest diameter and the axis perpendicular to it. This is the RECIST convention (Response Evaluation Criteria in Solid Tumors). Tens of thousands of these line pairs already sit in hospital archives. The usual way to turn them into training masks is a fixed image-processing heuristic such as GrabCut. That heuristic is noisy, it cannot be trained, and every later step inherits its mistakes. This note proposes a different route: treat the lines as sparse labels on a graph and let a learned model fill in the rest.

The graph spans many lesions at once. Its nodes are superpixels, meaning small groups of similar neighboring pixels, taken from each lesion's CT crop. A superpixel crossed by a RECIST line is labeled foreground. A superpixel far outside the ellipse the two lines define is labeled background. Everything in the uncertain ring between them is unlabeled. Those unlabeled nodes are classified by a relational graph convolutional network (R-GCN), a graph network that learns a separate message function for each kind of edge. The edge kinds here are qualitatively different: neighbors within one slice, neighbors in the adjacent slice, and look-alike superpixels in other lesions of the same morphological type. The bet is that evidence pooled across many lesions settles boundaries that a single lesion's pixels leave ambiguous. It is also a bet that typed edges are what allow this pooling to happen without blurring everything together.

Core Claim

Weakly supervised segmentation from RECIST lines can be posed as semi-supervised node classification. That is exactly the task R-GCNs were built for: the original paper, Modeling Relational Data with Graph Convolutional Networks, classified entities in partially labeled RDF graphs and introduced basis decomposition to share parameters across rare relations.

The claim is narrow and testable. A graph with cross-lesion edges and edge-type-specific transforms should beat the following three things on the uncertain boundary ring:

  1. A per-lesion network given the RECIST lines as input. This tests whether cross-lesion evidence matters.
  2. The same graph with all edges collapsed to one type. This tests whether edge typing matters.
  3. GrabCut-seeded pipelines. This tests whether learned propagation beats a fixed heuristic.

The gains should be largest exactly where GrabCut seeds are poor.

Relation to Prior Work

Two papers motivate the proposal by exposing the gap.

Weakly Supervised Lesion Co-Segmentation on Ct Scans does three things:

  • It turns RECIST lines into GrabCut pseudo-masks.
  • It groups lesions into 200 embedding clusters.
  • It trains a two-stream ("Siamese") network that compares pairs of lesions within a cluster.

This shows that sharing evidence across lesions helps. It also leaves two weaknesses in place: the sharing is strictly pairwise, and training still depends on the heuristic seed. We replace the pair with a cluster-wide graph and remove GrabCut from the training signal entirely.

DeepLesion supplies the data. It also documents that mined bookmarks are incomplete. This constrains our negative labels: background nodes are taken only from outside a margin around the annotated lesion's own ellipse, never from elsewhere in the scan.

The closest R-GCN work in medical imaging is MS-RGCN. It builds a two-relation graph over histology patches (spatial neighbors and cross-magnification containment) and trains it with slide-level labels only. It differs from this proposal in three ways:

  • It outputs one grade per slide, not a mask.
  • Its edges are purely geometric within a single image.
  • It never links separate cases.

We borrow its lesson that separate spatial and hierarchical kernels beat late fusion. We apply that lesson to dense output and to edges that cross cases.

Two papers supply methods we reuse:

  • Learning to Segment Medical Images with Scribble-Supervision Alone expands scribbles with random walks, then iteratively relabels using a conditional random field and Monte Carlo dropout uncertainty. We reuse its uncertainty masking for self-training rounds. We also use its setting as a second testbed: scribbles, like RECIST lines, label a sparse set of nodes.
  • HistoSegNet refines activation maps with a post-hoc conditional random field that is disconnected from feature learning. It represents the decoupled-refinement design we argue against.

One negative result shapes the design. Relational Graph Convolutional Networks Do Not Learn Sound Rules shows R-GCNs do not learn sound logical rules. We therefore claim only statistical label propagation, not symbolic reasoning about anatomy.

Research Program

Graph construction

For each DeepLesion lesion, we take a 3D crop around the bookmarked slice and its neighboring slices. We split it into superpixels and embed each superpixel with a frozen CNN encoder that is shared with all baselines. Each lesion also gets one "annotation node" that carries the line lengths and orientation. The relation types are:

Relation Connects
Intra-slice adjacency Neighboring superpixels in the same slice
Inter-slice adjacency Overlapping superpixels in adjacent slices
On long axis Annotation node to superpixels the long axis crosses
On short axis Annotation node to superpixels the short axis crosses
Cross-lesion similarity k nearest neighbors among superpixels of other lesions in the same morphological cluster, restricted to superpixels near each lesion's line endpoints

Cluster assignments follow Agarwal et al.'s 200-cluster scheme. Rare relations share parameters through basis decomposition.

Model and training

The layer is the standard R-GCN update with a self-loop. We train with:

  • cross-entropy on labeled nodes;
  • an extent penalty asking the soft foreground region's projection onto the long-axis direction to match the measured length;
  • two rounds of self-training in which only confident ring nodes, as judged by dropout uncertainty, become new labels.

Decoding and scale

Superpixel probabilities are painted back to pixels. Training runs in mini-batches of clusters, which keeps graphs at roughly cluster-sized subsets rather than the full corpus.

Analysis

We record which relations carry signal by measuring how much each relation's messages change ring-node predictions. We stratify results by lesion size and by GrabCut seed quality.

Baselines

Naive reference. A U-Net trained on RECIST-initialized GrabCut pseudo-masks, plus GrabCut masks evaluated directly. This is the "do nothing new" pipeline that the annotation-bottleneck literature starts from.

Strongest published method. The co-segmentation system of Weakly Supervised Lesion Co-Segmentation on Ct Scans, which reports a mean Dice of 90.3% on DeepLesion against 1,000 manual segmentations, using its channel-spatial attention block and D2 decoder. Because the file gives no code or split details, we will reimplement it on our splits and report both our reproduced number and theirs. Beating it means exceeding our reproduced figure on the same 1,000 masks with non-overlapping confidence intervals from a lesion-level bootstrap. It also means a clear gain on the subset where GrabCut seeds are worst.

Simple strong competitor. A per-lesion U-Net that receives the rendered RECIST lines as an extra input channel and is trained with our exact labeled-node losses and self-training. This tests whether conditioning on the annotation, rather than graph propagation, does the work.

Edge-typing ablation. The identical graph with all relations merged into one shared weight matrix, i.e. an isotropic graph convolutional network. A graph attention variant is included so that edge weighting alone is controlled for.

Cross-lesion ablation. The full relational model with cross-lesion edges removed.

Fairness. All learned baselines use the same frozen encoder, crops, epochs, and a matched hyperparameter search budget of equal trial count.

Why not just…?

Why not just feed the RECIST lines into a U-Net as an input channel? This is the most serious threat. It could plausibly match the full model on large, high-contrast lesions where the lines nearly determine the mask. Our proposal differs from it only in sharing evidence across lesions. That is why the cross-lesion ablation and this baseline are both in the design. If the U-Net matches the relational model on the uncertain ring, and especially on poor-seed lesions, we conclude that cross-lesion pooling adds nothing, and we report it as such.

Why not just refine pseudo-masks with a dense conditional random field? Agarwal et al. report that adding one raised precision but caused a "sharp reduction in recall." Can et al. found uncertainty masking, not the random field, drove their gains. We include a GrabCut-plus-random-field pipeline in the naive arm and expect the same recall loss on ring nodes.

Why not just use an isotropic graph network or attention on the same graph? MS-RGCN's gains over CLAM, TransMIL and other attention baselines (kappa up to 24.9% better on the Zurich set) suggest separate kernels matter. However, that evidence comes from slide grading, not segmentation. The edge-typing ablation settles the question directly for this task.

Why not just prompt a generic promptable segmentation model with the line endpoints? No reported result was found on Lacuna for such models on DeepLesion RECIST prompts. We will run one as an extra reference without tuning. If it already matches the strongest baseline, the contribution narrows to the poor-seed regime.

Proposed Experiments

Primary benchmark: DeepLesion

The primary benchmark is DeepLesion, scored on the 1,000 manual masks. We report three metrics:

  • mean Dice;
  • Dice restricted to the uncertain ring;
  • recall at fixed precision.

Each is stratified by lesion size and by GrabCut-versus-manual Dice, our measure of seed quality.

Decisive checks

The first decisive check is the pair of ablations.

  • If merging relations costs nothing, the relational claim fails.
  • If removing cross-lesion edges costs nothing, the pooling claim fails.

The second decisive check is the RECIST-channel U-Net on poor-seed lesions. This is where the proposal predicts its largest margin.

Transfer test: ACDC

A transfer test reuses the formulation on ACDC cardiac MRI. Scribbled superpixels are labeled nodes, and cross-patient edges link superpixels of similar slices within one anatomical class. The reference is Can et al.'s 88.0% Dice under scribbles versus 91.2% with full supervision. We ask whether the relational model narrows that 3.2-point gap under matched compute. A null result on ACDC with a positive result on DeepLesion would indicate the benefit is specific to lesions, which vary greatly in shape, rather than to sparse labels in general.