Shared-Space Expert Review for the Variants Where Predictors Disagree

Most genetic variants found in disease genes cannot yet be called harmful or harmless. They are "variants of uncertain significance." Machine-learning predictors now score them well on average. The hard residue is the variants where good predictors contradict each other, or contradict a lab assay. Those are exactly the cases that get dropped or reviewed slowly by one person at a desk. This note proposes a simple bet. Human experts add measurable value only on that residue, and they add it best when several of them can inspect the same mutated protein together. The protein would be a 3D model in a shared virtual space that every participant can grab, measure, and flex at once, with a fast physics model of its motions. Today the norm is one person driving a screen share while the others watch.

The point is not to build another virtual-reality viewer. The point is to run a controlled test of whether synchronous multi-user inspection beats cheaper alternatives on a benchmark with hidden ground truth. The alternatives are a stronger predictor, a panel of language-model agents, and independent experts whose votes are averaged. We use deep mutational scanning experiments as that ground truth: these measure the effect of nearly every possible substitution in a protein. If the shared space does not beat independent voting, we will report that. We will then ship the triage protocol, not the headset.

Core Claim

The claim has two parts. First, human structural review helps almost only on variants where predictors disagree. This mirrors TeraVR's finding that over 90% of immersive use went to low-signal, high-density regions (TeraVR). Second, on that subset, a team working in a shared space adds accuracy beyond what the same experts reach independently. The shared space supports simultaneous control, a blinded-then-unlocked measurement phase, and live views of residue flexibility.

What is new is the disagreement-targeted routing plus the head-to-head test against independent aggregation and against agent panels on ground-truth data. No paper in our search measures human adjudication accuracy on ProteinGym or ClinVar variants. We found no reported result on Lacuna for human or team accuracy on any ProteinGym subset.

Prior Work and Where This Sits

Two papers motivate the human-in-the-loop half.

  • VenusRAR (Rank-and-Reason) uses a panel of language-model "experts" to audit protein language model candidates before costly wet-lab tests. Its authors note that these agents struggle with raw 3D geometry and fall back on precomputed text descriptors. We keep its low-budget audit setting but replace synthetic agents with humans who manipulate the actual geometry. We keep the agents as a baseline.
  • The variant-resolution study (A scalable approach to resolving VUS) combines multiplexed assays with predictors and reclassifies 75% of existing uncertain variants in 40 genes. Its workflow leaves discordant cases unresolved. Those cases are our target population.

The predictive side supplies both the routing signal and the strongest baseline. TriFit (TriFit) fuses sequence embeddings, AlphaFold2 structure, and a Gaussian Network Model. That model treats the protein's alpha-carbons as beads joined by springs and yields cheap estimates of how much each residue flexes. TriFit reports that dynamics added +2.2 AUROC points while static structure added only +0.2. We take this as evidence that flexibility is worth showing to humans, so we reuse its Gaussian Network Model as the live dynamics layer. The model's harmonic assumption is a constraint: it cannot show large conformational changes, so we make no claim about those. MVP (MVP) is a second, conservation-heavy predictor whose disagreements with TriFit help define the routed subset.

The collaboration side supplies method, not evidence about molecules.

  • Specto (collaborative cyberspace VR oculometry) let three clinicians at distant sites measure retinal volumes in a shared space. It reports a 5.3 µm mean inter-grader difference across 327 measurements, at ≥90 Hz rendering. We borrow its blinded-measurement-then-unlock protocol.
  • The domain-expert study (CIVA expert study) found that scientists fall back on static screenshots and asked for shared provenance, meaning a record of who did what. We implement that record, but that study built no prototype.
  • TeraVR (TeraVR) shows immersive multi-user tracing catching structure that 2D tools missed. Its results concern neuron images, so they constrain our expectations rather than prove the molecular case.

Research Program

We build a small system with three parts.

The routing step. This step scores every substitution in a chosen protein with TriFit (retrained per its description if code is unavailable), MVP-style features, and the zero-shot models ProteinGym evaluates, including TranceptEVE and GEMME. A variant is marked "discordant" if the binarized calls of TriFit and the zero-shot ensemble disagree, or if TriFit's probability falls in a middle band. The band width is fixed in advance on a development set of assays.

The shared space. This is a game-engine client in which up to four users see the wild-type and mutant structures. Each user can rotate, slice, and measure distances independently. A Gaussian Network Model for a few-hundred-residue protein is a small eigenproblem, so it can be recomputed when a user swaps residues, and the client then animates per-residue flexibility and correlated motions. Every action is logged to a provenance graph keyed by user. Each variant follows a fixed sequence. Experts first give private calls and confidence. The views then unlock for a timed joint discussion. The team ends with a consensus call and confidence.

The analysis layer. This layer relates logged behavior to correct reversals of the predictor. The logged behavior covers which residues people measured, whether they opened the flexibility view, and who changed their mind.

Variants come from ProteinGym (ProteinGym) in two regimes:

  • Deep mutational scanning substitutions. The experimental score is hidden from participants and binarized at the top/bottom 30% as in TriFit.
  • The ClinVar clinical subset. We restrict to variants with definitive labels.

Novel clinical variants from case reports, such as HPRT1 p.Val35Ala, whose 3D modeling suggested a narrowed substrate pocket (HPRT1 case report), serve only as qualitative walkthroughs. They lack ground truth.

Baselines

The do-nothing reference. This is TriFit's call on each routed variant. TriFit reports 0.897 AUROC on binarized ProteinGym substitutions, ahead of Kermut at 0.864 and ProteinNPT at 0.844 (TriFit). It is the strongest existing method in the file. On the discordant subset it will be much weaker by construction, and we report its accuracy there exactly as measured. Every human condition sees TriFit's score, so beating it means correctly overturning it. The bar we set is a significant net gain: correct reversals minus incorrect reversals must be positive, with a confidence interval excluding zero. A gain in AUROC on the subset must also hold on the ClinVar portion.

An LLM agent panel in the style of VenusRAR. This panel receives the same predictor scores plus precomputed geometry descriptors. VenusRAR reports a 0.551 global Spearman correlation for its ranking stage and up to a 367% Top-5 hit-rate gain from its reasoning stage on ProteinGym-DMS99 (Rank-and-Reason). We run our reimplementation on the identical routed variants and give it the same per-variant information budget.

Independent experts with averaged votes. This is the key ablation. The same experts review on desktop PyMOL-style tools with the flexibility overlay, never interacting, and we aggregate their calls. It isolates what synchronous interaction adds over independent repetition.

A single-controller screen share. Here the team uses identical software but only one person can drive.

Two ablations of the shared space. One removes the flexibility view and one removes the blinded phase. They separate the contribution of the dynamics layer from the contribution of the anti-anchoring protocol.

Fairness controls. Every condition uses the same variants, the same wall-clock limit per variant, and the same predictor information. Condition order is counterbalanced across teams, and variant sets do not overlap between conditions.

Why not just…?

Why not just use a better predictor? If a model trained on more data closed the gap, humans would be unnecessary. TriFit's own ablations show the dominant remaining signal is dynamics, which the model already includes, yet discordant cases remain. The variant-resolution study shows that experimental-versus-computational discordance persists even with calibrated assays (VUS resolution). Our experiments settle this directly. If TriFit's accuracy on the routed subset matches the human conditions, the routing step reveals that humans add nothing, and we say so.

Why not just give LLM agents geometry tools? This is a real threat, because better tool access could fix the spatial weakness VenusRAR's authors admit. The agent baseline is therefore given computed distances, solvent exposure, and flexibility profiles, not only text summaries. If agents with these tools match teams, the cheaper answer wins.

Why not just have experts review independently and average their votes? This is the most serious alternative. Averaging independent judgments often beats discussion, which can spread one person's anchoring. Specto's blinded phase exists for this reason. The independent-vote condition is the decisive comparison. If it ties the shared space, we conclude that the collaboration modality is irrelevant, and that the contribution is the disagreement routing plus the flexibility overlay.

Why not just use static AlphaFold structures? The no-dynamics ablation tests this. TriFit's +2.2 versus +0.2 AUROC result suggests flexibility carries signal for models. Whether it carries signal for humans is unknown, and we measure it.

Proposed Experiments and Decisive Checks

We recruit 12 structural biologists and clinical geneticists in four teams of three. We draw roughly 600 routed variants from about 20 ProteinGym assays with AlphaFold structures, plus a ClinVar slice, and split them evenly across conditions with no overlap.

The primary metric is net correct reversals of TriFit on the routed subset. Secondary metrics are AUROC and expected calibration error of team confidence, minutes per variant, and simulator-sickness scores.

The low-budget design test mirrors VenusRAR. Each condition selects 30 variants per assay as "worth synthesizing," and we score the hit rate against hidden scan values.

Four results would be decisive:

  • The shared space beats independent voting on net reversals with a preregistered interval excluding zero.
  • That gain vanishes on a control sample of predictor-concordant variants, confirming the routing claim.
  • Removing the flexibility view reduces the gain.
  • Provenance logs show that correct reversals concentrate on variants where a measurement made by one expert changed another's call.

Failing the first check while passing the second reduces the contribution to the triage protocol, and we will report it that way.