Bispecific T-cell engagers are off-the-shelf antibody molecules with two grabbing arms: one binds CD3, the invariant signalling subunit of the T-cell receptor, and the other binds a molecule on the tumour surface. By holding both cells together, the drug forces an artificial immune synapse that bypasses the body's usual antigen-presentation pipeline and recruits the patient's existing polyclonal T cells directly to the tumour. The catch is that these drugs kill in unusual ways. IBI315, an anti-PD-1/Her2 bispecific tested in Her2-positive gastric cancer, does not just induce quiet programmed cell death — it triggers pyroptosis, a violent inflammatory rupture in which tumour cells swell, burst, and release IL-18, pulling waves of CD3+, CD4+, and CD8+ T cells into the lesion Anti-PD-1/Her2 Bispecific Antibody IBI315.... On a restaging CT scan, that influx of cells and fluid looks exactly like what every radiologist and every response criteria table has been trained to call progression: the lesion gets bigger.
At the same time, clinical AI has been moving from reading one scan at a time to reading sequences of scans — "longitudinal clinical reasoning" — because a finding only means something against its history. This proposal claims the intersection is not a curiosity but a concrete, unsolved diagnostic problem: the imaging trajectory of a bispecific-treated lesion encodes which cell-death program is running, and a longitudinal model conditioned on the engager's mechanism can decode it. Specifically, an inflammatory swell-then-shrink trajectory (pyroptosis plus immune infiltration) and a monotonic proliferative trajectory (true progression) are separable from serial volumetric imaging and serial laboratory values, and the same model can watch for the documented acquired-resistance trajectory — PD-1 upregulation on engager-activated T cells Membrane-Proximal Epitope Facilitates Efficient T Cell Synapse Formation by Anti-FcRH5/CD3... — before RECIST confirms failure.
Core Claim
No existing computational system distinguishes inflammatory pseudo-progression from true progression under bispecific engagement, or forecasts engager failure from evolving tumour state. The closest prior work on prediction, HR-Rad-Net, maps a single baseline CT to tumour-microenvironment phenotypes and response for anti-PD-1 monotherapy Non-invasive tumor microenvironment evaluation...; it says nothing about bispecifics, pyroptosis, or interval change. The closest prior work on interval change — DeltaNet, BioViL-T, CoCa-CXR, TiBiX — handles two-point 2D chest radiographs of pneumonia-like disease DeltaNet; Learning to Exploit Temporal Structure...; CoCa-CXR; TiBiX. Our claim is that closing this gap requires a mechanism-conditioned temporal model: a longitudinal encoder whose decomposition of interval change is gated by the known cytolytic program of the specific engager (pyroptosis-capable, like IBI315, versus synapse-forming and granzyme-mediated, like the anti-FcRH5/CD3 TDB). The mechanism token is the key ingredient; without it, a generic delta model sees only "lesion grew," which is precisely the ambiguity we need to break.
Research Program
We will build a therapy-conditioned longitudinal encoder over serial 3D contrast-enhanced CT, serial laboratory values (cytokines, inflammatory markers, circulating tumour DNA where available), and prior radiology reports. The architecture has four components, each adapted from a proven mechanism rather than invented wholesale.
First, a multi-epoch volumetric extension of regional cross-attention. CoCa-CXR showed that interval change is local and that letting each region of the current scan attend only to its spatial neighbourhood in the prior scan removes the noise that drowning-out subtle changes CoCa-CXR; we extend the masked attention to $k \geq 4$ timepoints over 3D volumes, staying registration-free in the spirit of BioViL-T, whose authors found rigid subtraction breaks under variable acquisition geometry Learning to Exploit Temporal Structure....
Second, the mechanism-conditioned two-channel decomposition — the proposal's central novel piece. Given the interval change features $\Delta x$ between consecutive restaging scans, the model decomposes them into an inflammatory channel $c_{\mathrm{inf}}$ (edema, immune infiltration, attenuation change without volumetric proliferative pattern) and a proliferative channel $c_{\mathrm{pro}}$ (coherent volumetric growth), with the split gated by a therapy token $\tau$ encoding the engager's known mechanism:
$$ (c_{\mathrm{inf}}, c_{\mathrm{pro}}) = g\big(\Delta x,\; \tau\big), \qquad \hat{y} = \mathrm{softmax}\big(W[c_{\mathrm{inf}}; c_{\mathrm{pro}}; x_t]\big) $$
where $x_t$ is the current-scan static feature, $\hat{y}$ is the response category (inflammatory pseudo-progression, stable disease, true progression), and $\tau$ is selected from a small discrete set of engager mechanism classes. The therapy token follows TiBiX's demonstration that a single learned token carrying the elapsed interval measurably improves temporal generation TiBiX — we are asking a token to carry mechanism rather than time.
Third, prior-report prompting with a textual gate, taken directly from BioViL-T's conditioning and DeltaNet's selective filtering of stale historical statements Learning to Exploit Temporal Structure...; DeltaNet, to keep generated follow-up reports grounded in the visual delta instead of the "unchanged" filler phrases that produce temporal hallucinations.
Fourth, an exhaustion-forecasting head. The anti-FcRH5/CD3 study documents that sustained CD3 engagement drives PD-1 upregulation on effector T cells, a defined acquired-resistance path Membrane-Proximal Epitope..., and CAR-T salvage work catalogues antigen-loss and exhaustion failure modes that clinical decision systems still treat as static baseline variables Exploring the mechanisms of CD19 CAR T-cell failure.... Our model will predict time-to-engager-failure from the trajectory of the decomposed channels plus laboratory streams — directly addressing the open problem that current longitudinal vision models ingest only pixels and text.
A second, equal deliverable is the benchmark itself. No reported result found on Lacuna for longitudinal imaging benchmarks in bispecific-engager-treated cohorts; every temporal benchmark in the file is thoracic radiography. We will assemble a retrospective multi-cohort resource — serial CT with RECIST/iRECIST-style annotations for Her2-positive gastric cancer, multiple myeloma, and DLBCL patients treated with engagers, drawing on the tumour types and targets established in the IBI315, anti-FcRH5/CD3, and DLBCL literature Anti-PD-1/Her2 Bispecific Antibody IBI315...; Membrane-Proximal Epitope...; Advances in biology, diagnosis and treatment of DLBCL — with lesion-level response adjudicated by the gold-standard outcome a retrospective design allows: whether a lesion that grew at cycle 2 subsequently shrank by cycle 6 (inflammatory) or kept growing (progressive). Lesion measurement tooling and annotation protocols will follow the RECIST-annotation line of work Recist-Net, and radiogenomic anchoring of the pyroptosis axis (GSDMB expression versus imaging phenotype) will use matched imaging-transcriptomic data in the TCGA/TCIA-STAD resource, where the IBI315 study already validated GSDMB expression Anti-PD-1/Her2 Bispecific Antibody IBI315....
Baselines
The naive reference point is standard RECIST 1.1 practice: unidimensional diameter measurement at each restaging, with a 20% increase called progression. This is what actually happens in clinics today, and its documented weaknesses define the floor we must clear — blinded double reads show substantial inter-observer variability and baseline lesion-selection ambiguity RECIST 1.1 assessments variability...; RECIST 1.1 and lesion selection..., and diameter is structurally blind to intratumoral biology that radiomics demonstrably captures Non-invasive tumor microenvironment evaluation.... We will simulate the RECIST read from lesion annotations and report its pseudo-progression misclassification rate directly.
The strongest existing competitor is HR-Rad-Net's deep learning radiomics signature (DLRS) Non-invasive tumor microenvironment evaluation... — correcting the id: Non-invasive tumor microenvironment evaluation and treatment response prediction in gastric cancer using deep learning radiomics. It is the only prior model that links CT appearance to immune-microenvironment state and therapy response, reporting a tumour-microenvironment classification AUC of 0.909 to 0.937 across internal and external cohorts, an objective response rate of 57.8% versus 14.2% for high versus low DLRS under anti-PD-1 therapy, and an AUC of 0.783 for predicting immunotherapy response when combined with PD-L1 combined positive score. To make the comparison fair, we will re-implement HR-Rad-Net faithfully, run it on exactly our volumes at each restaging timepoint (giving it the same multi-timepoint information in the only form its architecture accepts, one scan at a time), fuse its PD-L1-CPS-style clinical covariates exactly as published, and give it the same tuning budget and training folds. Beating it means a statistically significant improvement in AUC for true-versus-pseudo progression at the first restaging scan, and — the more damning test — showing that its per-timepoint predictions cannot separate lesions that swell-then-shrink from lesions that swell-and-keep-growing, since its training signal was baseline-only.
The closest temporal competitor is the two-point regional cross-attention of CoCa-CXR CoCa-CXR, which reports 65.0% progression-classification accuracy on MS-CXR-T, outperforming BioViL-T by 4.8%; BioViL-T reports 87.7% on temporal sentence paraphrase/contradiction over the same data Learning to Exploit Temporal Structure.... We will port both to 3D multi-timepoint inputs (2-point window for their native design, plus a best-effort multi-timepoint extension), same data and compute. Finally, the ablation that isolates our key ingredient is the full architecture with the therapy token $\tau$ removed and the two-channel decomposition replaced by a single undifferentiated delta channel; if pseudo-progression discrimination survives that ablation, the mechanism-conditioning claim is dead and we will report it as such.
Why not just…?
Why not just run a strong static classifier at each timepoint? This is the HR-Rad-Net baseline above, and it is a real threat — its microenvironment AUCs (0.909–0.937) are strong. But the discriminating signal we are after is intrinsically trajectory-shaped: inflammation and proliferation produce nearly identical single-scan appearances (enlarged, denser lesion) and diverge only in what happens next. Single-image-proficient vision-language models demonstrably degrade on comparative progression tasks Medical Large Vision Language Models with Multi-Image Visual Ability, and high static accuracy can mask missing process reasoning — the "evaluation illusion" documented by Neural-MedBench Beyond Classification Accuracy.... Still, the comparison is run head-to-head, and if per-timepoint HR-Rad-Net matches our pseudo-progression AUC, the temporal claim fails honestly.
Why not just subtract the scans (or use RECIST diameters) to detect change? Naive feature or pixel subtraction is defeated by acquisition variability — precisely why BioViL-T was built registration-free — and by background anatomical difference drowning the signal, which is why CoCa-CXR's neighbourhood masks bought their 4.8% margin CoCa-CXR; knowledge-aware difference graphs were needed for the same reason in difference-aware VQA Expert Knowledge-Aware Image Difference Graph.... RECIST diameters are worse than subtraction here, not better: under a pyroptosis-inducing engager the diameter increases because of the very inflammation that predicts response, so the cheap method is not merely imprecise but sign-flipped. We quantify this flip as the RECIST baseline's misclassification rate.
Why not just prompt a large language model with the series of radiology reports? Text-only longitudinal reasoning is exactly where the documented failure mode lives: models copy clinical clichés like "unchanged" and "resolved" without visual grounding, and TiBiX showed that grounding generation in the actual prior scans is what produced its 1.5x BLEU-4 improvement TiBiX; Learning to Exploit Temporal Structure.... Clinical reality also spans non-text evidence streams that text prompting cannot ingest ClinSeekAgent. We will include a prompted-LLM-over-reports baseline for completeness, and expect it to hallucinate interval claims — measuring that rate is itself a secondary contribution.
Proposed Experiments
Experiment 1 is an architecture sanity check on existing public benchmarks so the encoder is not validated only on data we built ourselves. On MS-CXR-T progression classification, the bar is CoCa-CXR's 65.0% accuracy; on MIMIC-CXR report generation, DeltaNet's clinical-efficacy F1 of 0.406 (versus 0.144 for its retrieve-and-filter baseline) is the reference point DeltaNet; CoCa-CXR. We do not need to beat these on their home turf — that is not our question — but the multi-timepoint encoder must be within a few points, or the port to 3D is suspect.
Experiment 2 is the headline test: on the assembled bispecific cohort, discriminate inflammatory pseudo-progression from true progression at the first restaging scan, with ground truth adjudicated by the subsequent trajectory. Metrics: lesion-level and patient-level AUC, and sensitivity at 90% specificity for calling true progression (the operationally relevant point, since falsely stopping an effective engager is the error the field currently makes). Baselines as specified above. The pre-specified success criterion is a meaningful margin over the RECIST read and over per-timepoint HR-Rad-Net, with confidence intervals from patient-level bootstrap.
Experiment 3 is the exhaustion forecast: predict engager failure (PD-1-exhaustion or antigen-loss trajectory) before RECIST-confirmed progression, evaluated as AUROC at each cycle and median lead time in weeks. This is the component that uses the laboratory streams; no prior system in the file attempts it, and no reported result found on Lacuna for forecasting engager failure from longitudinal multimodal data, so we set the naive reference as "RECIST progression confirmed at the same cycle."
Experiment 4 is the decisive ablation and the mechanistic grounding. The ablation drops the therapy token and the two-channel split; we predict pseudo-progression AUC falls materially, and if it does not, we conclude mechanism-conditioning is unnecessary and report that. For grounding, we check whether the inflammatory channel activates in GSDMB-high settings using the matched imaging-transcriptomic gastric cancer data in TCGA/TCIA-STAD, where the IBI315 study already established GSDMB expression and left open whether GSDMB is a better predictor than PD-L1 CPS Anti-PD-1/Her2 Bispecific Antibody IBI315... — our model offers a non-invasive, serial way to ask that question that a single immunohistochemistry stain cannot.
The main risk is data: the benchmark does not exist and must be built, and a proposal whose evaluation rests on its own dataset invites circularity. We mitigate this by (a) adjudicating ground truth only from future scans the model never sees at decision time, (b) reporting all baselines on identical folds, and (c) anchoring the mechanism claims on public radiogenomic and preclinical evidence rather than on our private labels alone. If cohort assembly yields fewer than ~150 engager-treated patients with usable serial CT, we will fall back to a lesion-level design and pre-define that reduction before unblinding.
References
- Anti‐PD‐1/Her2 Bispecific Antibody IBI315 Enhances the Treatment Effect of Her2‐Positive Gastric Cancer through Gasdermin B‐Cleavage Induced Pyroptosis
- Membrane-Proximal Epitope Facilitates Efficient T Cell Synapse Formation by Anti-FcRH5/CD3 and Is a Requirement for Myeloma Cell Killing
- Non-invasive tumor microenvironment evaluation and treatment response prediction in gastric cancer using deep learning radiomics
- DeltaNet: Conditional Medical Report Generation for COVID-19 Diagnosis
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
- CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding
- TiBiX: Leveraging Temporal Information for Bidirectional X-Ray and Report Generation
- Exploring the mechanisms of CD19 CAR T-cell failure and salvage strategies in B-cell lymphoma
- Advances in biology, diagnosis and treatment of DLBCL
- Recist-Net: Lesion Detection Via Grouping Keypoints On Recist-Based Annotation
- RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads
- RECIST 1.1 and lesion selection: How to deal with ambiguity at baseline?
- Medical Large Vision Language Models with Multi-image Visual Ability
- Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks
- Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering
- ClinSeekAgent: Automating Multi-modal Evidence Seeking for Agentic Clinical Reasoning