Watching Bystander Killing: Few-Shot Cell-Action Recognition for Closed-Loop Safety Switches

Engineered immune cells can now be built with a drug-inducible on/off switch, but nobody can see what those cells are actually doing at the moment the switch matters. When a CAR T cell (an immune cell reprogrammed to attack a chosen target) engages one tumor cell, it often kills neighboring tumor cells that lack the target entirely — the bystander effect, the main reason these therapies work against heterogeneous tumors at all. Yet bystander killing is measured today only as an aggregate number: a tumor shrinks, or a viability assay dims, ten days later. We propose to watch it happen. The plan is to adapt few-shot action recognition — techniques for learning new action classes from one to five labeled video clips — to live-cell microscopy, so that rare, brief cell-contact events (approach, synapse formation, lethal hit, detachment, death of an untargeted neighbor) can be recognized from a handful of expert-labeled examples. The resulting single-cell measurements then serve two purposes: they quantify how bystander killing depends on the fraction of target-positive cells in a mixed tumor, and they become the feedback signal for an automatic dosing controller that titrates a drug-inducible safety switch in the effector cells.

The motivation is a control problem with no sensor. Inducible switches exist — a doxycycline-gated CAR that turns on and off over roughly a day, a suicide gene, an antibody-triggered depletion handle — but they are operated open-loop by clinicians reacting to systemic symptoms hours or days late Synthetic biology approaches.... Meanwhile, the events that decide safety and efficacy — whether an effector cell serially contacts antigen-negative neighbors or saturates a region with a diffusible kill signal — play out over minutes under a microscope. Tumor heterogeneity sets the stakes: bystander killing in a cold pancreatic tumor model only clears the tumor when at least 83% of cells carry the target antigen, dropping to 29% only with adjuvant interleukin-12 Tumor neoantigen heterogeneity.... That threshold is exactly the kind of quantity a perception layer could measure and a controller could exploit — but no instrument currently measures it at the level of individual killing events.

Core Claim

Cell–cell killing behaviors, though non-rigid and skeleton-free, can be decomposed into a small vocabulary of contact-state primitives — a cellular analogue of the Therblig primitives used for human manipulation video — and recognized from 1–5 labeled examples by adapting alignment-based few-shot action recognition to region- and contact-based (rather than skeletal) representations. This recognition layer yields the first dynamic, per-event measurement of bystander killing in antigen-mixed co-cultures, and those measurements are sufficient feedback to drive dose-responsive actuation of an inducible switch. We deliberately scope the control claim to what switch kinetics permit: a transcriptional switch cannot arrest an individual lethal hit, but a recognition-driven dosing policy can steer population-level behavior over hours, trading bystander spread against clearance rate.

Research Program

Three components, each feeding the next.

First, a primitive vocabulary and few-shot recognizer for effector–target interactions. Therbligs in Action decomposes human actions into contact primitives (Grasp, Move, Release) and enforces physical ordering rules as differentiable losses, gaining +6.5% relative on action recognition over EPIC Kitchens 100; we port this idea to cells, where the natural primitives are contact-state-based: Approach, Adhesion/Tether, Synapse Engagement, Lethal-Hit Delivery, Detachment, Engulfment, and Bystander-Neighbor Death. Cells have no joints, so the representation is built from segmentation masks, overlap and contact graphs, and deformation descriptors rather than skeletons — directly answering the known failure of skeleton-based few-shot models on amoeboid motion noted in the live-cell vision literature. For the few-shot machinery we build on the two strongest published designs: VIM, which maximizes mutual information via task-adaptive spatiotemporal sampling and query–support alignment (86.2% 1-shot on UCF101), and TARN, whose segment-by-segment attention suits variable-duration cell contacts; CMN supplies the memory design for accumulating primitive statistics. Differentiable rule constraints ("no Detachment before Synapse Engagement"; "Bystander-Neighbor Death must follow a Lethal Hit within radius $r$ and time $\tau$") regularize the recognizer, following the Therbligs result that such rules help most in low-data regimes.

Second, a benchmark for bystander behavior. We extend the Cell Behavior Video Classification Challenge protocol — intravital/time-lapse patches of 40 × 40 µm, 50 × 50 pixels, 20 frames over 19 minutes, with a 70/10/20 split — from its single binary task (did the T cell turn?) to multi-phase interaction labeling in antigen-mixed co-cultures CBVCC. Co-cultures mix antigen-high and antigen-low lines at controlled fractions mirroring the clonal ratios in the pancreatic bystander study (29%, 68%, 83%): HER2-high NCI-N87 with HER2-low BxPC-3 biparatopic HER2 ADC, or high-GD2 DIPG line TT150630 with its low-GD2 counterparts Anti-tumor efficacy of anti-GD2 CAR NK-92.... Effectors carry the Tet-On inducible CAR architecture of Doxycycline Inducible CAR T Cells Targeting CD147, whose measured kinetics (expression peaking 24 h after doxycycline, clearing within 48 h of withdrawal; ~50% lysis at 10:1 effector:target when induced, negligible when uninduced) parameterize the controller. Tracking infrastructure follows TIAM, whose multi-channel motility analysis of T cells shows how to extract contact area, arrest, and speed features. The recognizer outputs two summary statistics per field of view:

$$B = \frac{\text{deaths of antigen-negative cells}}{\text{lethal hits on antigen-positive cells}}, \qquad R = \text{median distance of bystander death from last synapse},$$

the bystander ratio and the bystander kill radius, neither of which any endpoint assay can produce.

Third, a closed-loop dosing policy. A simple model-predictive controller maps rolling-window estimates of $B$, $R$, and the antigen-positive kill rate to doxycycline dose, using the measured 24 h/48 h switch dynamics as the actuator model. The control objective is to keep the kill rate on antigen-positive cells above a set fraction of the open-loop rate while holding bystander deaths in healthy-parenchyma patches near zero. We are explicit that this steers a population over hours; it cannot stop a single lethal hit already in progress, and we design and evaluate it accordingly.

Baselines

The do-nothing reference is the current standard in both halves of the problem: constant open-loop doxycycline dosing of the Tet-On CAR, which by the measurements in Zhang et al. yields full effector activity with no modulation, and bulk endpoint viability/bioluminescence readouts, which is exactly how the pancreatic bystander study quantified clearance thresholds Tumor neoantigen heterogeneity.... Beating this reference means the closed-loop policy reduces bystander deaths in healthy-cell patches by a pre-registered margin while retaining at least 80% of open-loop antigen-positive killing.

The strongest existing method for classifying cell behavior from video is the CBVCC winner, Team GIMR: an attention-based 1D CNN operating on automated tracking features, which scored 0.922 (AUC 0.944) on the challenge's binary turning task CBVCC. This is the key competitor because it represents the "tracking plus learned classifier" pipeline the bio-imaging community would reach for first. We will run it on our multi-phase benchmark with the same videos, the same annotation budget, and the same tuning budget (fixed-count random search over hyperparameters for every method), and additionally feed it perfect manual tracks, since CBVCC reports that manual tracks lift classification to nearly 100% while automated tracks degrade it — that gap is precisely where dense mixed co-cultures will stress it. Beating it means higher macro-averaged accuracy on the multi-phase labels at matched annotation cost, particularly in dense fields where tracking identity switches occur.

The strongest few-shot competitor is VIM Few-shot Action Recognition via Intra- and Inter-Video Information Maximization, ported to our cell videos with its sampling and alignment modules intact; its 86.2% 1-shot UCF101 and 61.1% 1-shot HMDB51 results set the expected caliber, and TARN (66.55% 1-shot on Kinetics) and CMN (60.5% 1-shot, 78.9% 5-shot on Kinetics-100) run as secondary few-shot references, all retrained on our benchmark under identical episode protocols.

Two ablations isolate the proposal's key ingredients: the recognizer with primitive decomposition but without the differentiable ordering rules (the Therbligs paper attributes +6.5% relative recognition gain to exactly these rules on EPIC Kitchens 100, so we test whether that transfers to cell contact logic), and the recognizer fed raw tracking-feature sequences instead of contact-primitive representations, which cleanly separates "primitives" from "few-shot machinery."

Why not just…?

Why not just track cells and threshold on kinematics? This is the most serious alternative, and the CBVCC result gives it real force: tracking plus a small learned classifier already reached 0.922 on cell behavior classification CBVCC. But that benchmark was a single binary decision on sparsely arranged cells; our task requires distinguishing a bystander death from a mitotic arrest or a detachment in dense, touching mixtures, where CBVCC itself documents that tracking identity switches between contacting cells corrupt downstream classification, and where end-to-end video models failed catastrophically. Rather than argue around it, we run the tracking pipeline head-to-head on our benchmark with manual tracks as its ceiling; if it matches primitive-based recognition there, the honest conclusion is that the primitive layer is unnecessary for perception and only the control contribution stands.

Why not just fine-tune a large pretrained video model? The evidence rules this out at our data scale: on 300 video patches, the best off-the-shelf end-to-end model in CBVCC (SlowOnly-R101) overfit to a score of 0 CBVCC, and video transformers fared no better. Expert annotation of rare killing events is the bottleneck, so the few-shot setting is not a stylistic choice but the actual experimental constraint; we will still fine-tune the strongest video backbone under the same annotation budget so the comparison is empirical, not rhetorical.

Why not just skip video and predict response from omics? Models like DeepCDR (Pearson 0.923 for IC50 prediction across GDSC) and THERAPI (AUROC 0.775 transferring preclinical response to TCGA patients) predict bulk sensitivity labels; they cannot resolve whether killing of antigen-negative neighbors happens by serial contact or a diffusible field, which is the variable our controller must regulate DeepCDR; THERAPI. Even MMCL-CDR, which adds static morphology images to omics, uses single frames and reports gains of at least 1.5% AUC on bulk labels — a different question, answered at a different resolution MMCL-CDR. The decisive check is whether the per-event bystander statistics our recognizer extracts actually predict the in vitro clonal-fraction clearance threshold; if they do not add value over endpoint assays, the perception claim fails on its own terms.

Why not just use a kill switch triggered by clinical symptoms, or a stronger diffusible payload instead? Symptom-triggered ablation is what exists today and it is irreversible and late Synthetic biology approaches.... Turning bystander killing entirely over to membrane-permeable payloads is the pharmacological alternative, but payload diffusion cannot distinguish an antigen-negative cancer cell from an adjacent antigen-negative healthy cell — the mechanism behind interstitial lung disease with such ADCs biparatopic HER2 ADC — and immune bystander clearance fails outright below high clonal fractions without microenvironment remodeling Tumor neoantigen heterogeneity.... A tunable, perception-gated effector is the only option in this set whose bystander footprint can be steered at all; whether steering it is worth the machinery is exactly what the closed-loop versus open-loop experiment decides.

Proposed Experiments and Decisive Checks

Experiment one establishes perception. On the mixed co-culture benchmark (clonal fractions 29%, 68%, 83%; CBVCC-style patches and splits; 5-way 1-shot and 5-shot episodes), the primitive-based recognizer competes against the GIMR-style tracking pipeline, the ported VIM/TARN/CMN few-shot models, and the two ablations, all under matched annotation and tuning budgets. Success is few-shot accuracy clearly above the tracking baseline in dense fields, with reference calibrations from the human-video literature (TARN 66.55% 1-shot on Kinetics; CMN 78.9% 5-shot on Kinetics-100; VIM 86.2% 1-shot on UCF101). No reported result found on Lacuna for few-shot action recognition on live-cell interaction videos, which is the point.

Experiment two validates the measurements. Per-event $B$ and $R$ statistics, aggregated per clonal fraction, must predict which mixtures cross the clearance threshold — the in vitro analogue of the 83%/29% in vivo thresholds in the pancreatic model Tumor neoantigen heterogeneity.... The assay-level confirmation is the endpoint viability measurement; the new content is that we can say whether threshold-crossing mixtures kill bystanders serially (small $R$, $B$ bounded per engagement) or diffusively (large $R$), which endpoint assays cannot distinguish and which determines whether a contact-blocking or activation-damping control action is appropriate.

Experiment three closes the loop. In imaging chambers where healthy-parenchyma patches sit adjacent to tumor patches, the recognition-driven dosing policy runs against constant open-loop dosing and against a timer-based schedule matched for total doxycycline exposure. Success is a pre-registered reduction in bystander deaths in healthy patches while retaining at least 80% of open-loop antigen-positive killing, using the Tet-On kinetics (24 h to peak, 48 h to clear, ~50% lysis at 10:1 when induced) as the actuator model Zhang et al.. If open-loop dosing matches the closed-loop policy, the honest conclusion is that perception adds measurement value but not control value at current switch speeds — a finding that would itself redirect the field toward faster actuators, for which the modular adapter approach of Märkl et al. is the natural next candidate.