Locus Slots: Tracking Who Is Pointed at in Signing Space with Cheap, Adaptive Refinement

Signed languages keep track of people and things by placing them in the air. A signer introduces "my sister" and sets her at a spot to the left. Later, pointing to that spot means "she," with no word for "she" needed. Linguists call these spots loci. Machines handle this badly. The main computational study of signed coreference, meaning the task of deciding which mentions refer to the same entity, resolves "I" and "you" at 92.2 F1 because those are anchored to real bodies. For third-person pointing, where the target is an empty point in the air, it scores only 26.9 F1 (Signed Coreference Resolution). The cost shows up downstream. When fluent Deaf signers were shown sentence clips cut out of their discourse, a third of the clips could not be translated, because the antecedents had been set up earlier (Reconsidering Sentence-Level Sign Language Translation).

We propose a single transformer that reads 3D body motion and text in one input sequence. It keeps a small, persistent set of locus slots. Each slot is a memory vector that holds both a 3D position in signing space and the identity of the entity parked there. A pointing gesture is resolved by scoring it against these slots. When the choice is ambiguous, the model re-runs a few cheap update steps over the slots only, never over the full video. This yields the accuracy benefit of multi-pass coreference refinement at a small fraction of the latency of text-based self-critique. The test case is also a clean instance of abstract grounding. Signers give absent people, institutions and ideas a location with no visual appearance, so a locus is a spatial stand-in for something that cannot be seen.

Core Claim

Third-person pointing fails for three reasons that can be separated. Current systems work from 2D hand coordinates and gold glosses, so nearby loci collapse together. They have no memory that persists across sentences and can be overwritten when a signer reassigns a locus to a new entity. And they make one-shot decisions where global consistency needs several passes. We claim that an explicit slot memory with 3D position embeddings, plus refinement that runs only on uncertain mentions and is distilled from a fixed-depth teacher, will substantially raise third-person INDEX F1 on DGS-Coref over the multigraph baseline. We further claim it will match the accuracy of fixed-depth iterative refinement while adding only milliseconds per mention, rather than the doubling of response time seen with text critique loops.

The claim that matters is not "put 3D into a transformer." It is that a small, rewritable locus memory is the right place both to store discourse state and to spend refinement compute. We also name the result that would falsify this: a well-engineered 3D geometric heuristic matching the learned model.

Relation to Prior Work

Several papers motivate the gap. Signed Coreference Resolution builds an unsupervised multigraph whose edges come from 11 heuristics, such as 2D hand distance and the number of intervening signs, and then clusters mentions. It reports the concrete-versus-unanchored split above. It also names locus reassignment as unhandled, and our slot overwrite mechanism targets exactly that. Reconsidering Sentence-Level Sign Language Translation shows that missing discourse context makes a third of clips untranslatable, with classifiers accounting for 47% of context failures. Including Signed Languages in Natural Language Processing frames signing space as a persistent database and documents the collapse from studio to natural data: 22.17 BLEU on PHOENIX versus 3.2 on the Public DGS Corpus.

Other papers supply methods we reuse.

  • 3D position embeddings. 3D Aware Region Prompted Vision Language Model adds depth-derived 3D position embeddings to visual tokens in one shared stream. It works only on static rooms and reports that dense spatial embeddings hurt text-heavy tasks. We borrow the position embedding, but apply it to moving hands and keep it confined to slot and pose tokens to limit that interference.
  • Motion tokens in a language model. Large Sign Language Models shows that SMPL-X motion (a standard 3D body-and-hand mesh model) can be discretized and projected into a language model's token space. It evaluates only isolated sentences. We use a similar motion front end but add discourse memory.
  • Refinement without repeated decoding. Sign Language Translation with Iterative Prototype refines a latent vector for a fixed three steps before decoding once. Graph Refinement for Coreference Resolution refines coreference links for four passes to reach 80.5 average F1 on CoNLL 2012. Both use fixed schedules and neither models space. We refine over slots with an input-dependent number of steps.
  • Collapsing loops into one pass. Meta-Reflection distills multi-pass reflection into a single pass (text-only, 951 ms to 246 ms first-token latency on ECID). We reuse the distill-the-iterative-teacher idea for multimodal slot refinement.

Two papers constrain the design. Volcano shows that critique-revise-decide loops help hallucination but raise latency from 2.7 s to 5.8 s. That rules out text-level revision for streaming sign. Lost in Translation, Found in Context shows that passing previous sentences as text helps translation (40.3 BLEURT on BOBSL), but reports that errors in earlier sentences propagate forward. The closest existing work is therefore the multigraph plus IP-SLT and G2GT. None of them combines a learned, rewritable spatial memory with adaptive-depth refinement.

Research Program

Front end. Each video frame is turned into monocular 3D pose, specifically SMPL-X body and hand parameters, by fitting as done for SignAvatars. Short motion windows become tokens through a motion encoder in the style of Large Sign Language Models. For every detected pointing or agreement-verb mention, the system computes a pointing ray: an origin at the fingertip or verb endpoint, and a direction.

Shared backbone. One transformer receives motion tokens for a sliding window, text tokens (the partial translation or glosses when available), and $K$ locus slot tokens carried over from the previous window, with $K$ around 8. Each slot $k$ holds a content vector $c_k$ and a 3D position $p_k$ in a signer-centred frame. The position enters as an added 3D position embedding. Only the slots cross window boundaries, so memory cost grows with $K$, not with discourse length.

Resolution. A mention $m$ has a query vector $q_m$ and a pointing ray $r_m$. It is assigned to a slot, or to a "new entity" option, by

$$ s_{mk} = q_m^\top c_k - \lambda \, d(r_m, p_k)^2 , $$

where $d(r_m, p_k)$ is the perpendicular distance from slot position $p_k$ to the pointing ray and $\lambda$ is a learned scale. Establishing mentions, such as a fingerspelled name followed by placement, write to a slot. Writing to an occupied slot while establishing a new entity counts as a reassignment, and the old content is evicted.

Adaptive refinement. After the first pass, we compute the entropy $H_m$ of the softmax over $s_{m\cdot}$ for each mention. Only mentions with $H_m > \tau$ trigger another update step. That step is a small shared-weight block of attention between slots and uncertain mentions, capped at $T_{\max}=4$ steps to match G2GT's depth. Everything else in the window is frozen. Training has two parts. A teacher always runs $T_{\max}$ steps, and the student's early-exit assignments are pulled toward the teacher's final ones with a distillation loss, following IP-SLT and Meta-Reflection. The threshold $\tau$ is swept to trace an accuracy-latency curve.

Supervision. DGS-Coref provides coreference clusters but is small, so we use two sources of supervision. First, cluster labels under cross-validation. Second, self-supervised locus consistency on the unlabeled Public DGS Corpus. That loss rewards repeated pointing at a stable region to reuse a slot unless an establishing sign intervenes. It is weak and could teach "always reuse," so it is ablated. We also add a hand-annotated reassignment subset of the DGS-Coref test discourses. For the abstract-grounding lens, we tag each gold entity as a present person, an absent concrete referent, or an abstract referent (institution, idea, group).

Baselines

Unsupervised multigraph (reference point). The first reference is the unsupervised multigraph of Signed Coreference Resolution, run on the same DGS-Coref discourses and gold mention spans. It reports 50.54 mean F1, 92.2 F1 on I/YOU and 26.9 F1 on INDEX. It is the only published system for the task. Beating it means a clear gain on INDEX F1 without losing I/YOU F1. A naive "do nothing" floor is also included: link each third-person pointing to the most recent third-person entity. This tells us how much of the 26.9 comes from recency alone.

3D multigraph (strongest cheap competitor). The multigraph's 2D hand-distance heuristic is replaced with ray-to-region distance computed from our 3D pose. Its other heuristics and the clustering are tuned on the same development folds with the same search budget we give our threshold $\tau$. This isolates whether 3D geometry alone explains any gain.

Text-context model. This is the approach of Lost in Translation, Found in Context, whose best reported number is 40.3 BLEURT on BOBSL. We re-implement it on our backbone size: previous-sentence translations and pseudo-glosses are given as text, and coreference is read from an attention-based linker. It receives the same pretraining data and fine-tuning steps.

Refinement variants. The key ablations use the same trained model with refinement at a fixed $T=1$ and at a fixed $T=4$, the G2GT-style schedule. A further variant removes the slot memory and uses plain long-context attention over the whole discourse.

Latency reference. A Volcano-style critique-and-revise loop on our backbone serves as the latency reference. Volcano's own figures are 5.8 s against 2.7 s, but on visual question answering rather than signing. No reported latency result for signed coreference was found on Lacuna, so all latencies are measured on the same GPU.

Why not just…?

Why not just feed previous sentences as text to a large language model? It demonstrably helps translation (Lost in Translation, Found in Context). But text says nothing about where an entity was placed. The same authors report that errors cascade, and Reconsidering Sentence-Level Sign Language Translation found that humans given full prior English text still struggled when spatial setup was missing. The text-context baseline is run on identical data. Our prediction is that it closes much of the gap on discourses with one active third-person referent but falls behind when two or more loci are active. We report results stratified by number of active loci, so the question is settled by data.

Why not just use better geometry in the existing heuristic graph? This is the real threat. If 2D projection is the whole problem, the 3D multigraph baseline may recover most of the INDEX gap at negligible cost. We treat that outcome as a finding rather than a failure to explain away. If the 3D multigraph comes within a few F1 of our model overall, the contribution shrinks to the cases geometry cannot handle: reassignment, and loci closer together than the pose-estimation error. We therefore report those subsets separately. If the learned model does not win there either, we would drop the learned slot memory and publish the geometric result. The in-the-wild fragility of pipeline systems (Auslan-Daily, 2.72 BLEU-4 on News) suggests heuristics will degrade on natural data, but that is translation evidence, not coreference evidence.

Why not just run a fixed number of refinement passes, or a text critique loop? Fixed $T=4$ may simply be good enough, and it is cheap when it runs over only $K$ slots. The adaptive gate is justified only if it keeps $T=4$ accuracy while cutting mean added latency. That is exactly the fixed-versus-adaptive comparison above. Text critique loops add full decoding passes, which Volcano measured at more than double the response time. We expect them to be dominated on the latency axis, and we measure whether they gain anything on accuracy.

Why not just use one long-context transformer with no explicit memory? Attention over the full discourse can in principle find antecedents. Its cost, however, grows with the square of the sequence length, and it has no explicit way to overwrite a reassigned locus. The no-slot ablation, given the same compute, tests whether the slots add accuracy on reassignment cases or only save memory.

Proposed Experiments and Decisive Checks

Main coreference result. We report DGS-Coref under cross-validation, using CoNLL-style average F1, and break it down by I/YOU, INDEX and agreement verbs. Results are given both with gold mention spans (comparable to the 50.54 / 26.9 figures) and with predicted spans. The first decisive check is whether INDEX F1 rises substantially above 26.9 and above the 3D multigraph. Beating only the 2D original would show that geometry, not memory, mattered.

Memory and referent-type subsets. The second check is the reassignment subset, where every baseline without rewritable memory should fail systematically. The abstract-grounding analysis compares the three referent types. If slots help abstract referents as much as concrete absent ones, a spatial proxy is sufficient grounding for coreference, since appearance was never available. If abstract referents lag, the model is quietly relying on lexical cues from the establishing sign.

Accuracy-latency trade-off. The third check is the curve traced by sweeping $\tau$. It plots INDEX F1 against added milliseconds per mention and against streaming lag. It is compared with fixed $T=1$, fixed $T=4$ and the critique loop. Success means matching fixed-$T=4$ F1 within noise at clearly lower mean cost, with most mentions exiting after one step.

Downstream translation. We attach a decoder and test whether better locus tracking improves translation. On BOBSL, the reference is 40.3 BLEURT from Lost in Translation, Found in Context. On How2Sign, we evaluate specifically on the context-dependent clips identified by Reconsidering Sentence-Level Sign Language Translation. No reported result was found on Lacuna for automatic contextual translation on How2Sign, so there we compare against our own text-context re-implementation.

Interference check. Because 3D Aware Region Prompted Vision Language Model found that spatial embeddings hurt text-heavy tasks, we also measure translation fluency with and without slot tokens. This checks whether the spatial memory competes with language generation.