Pointing and Speech as Each Other's Labels: Cross-Modal Co-Training for Deictic Reference Resolution

When someone says "hand me that" while pointing, a machine has to work out which object they mean. Current systems learn this from thousands of hand-drawn target masks. This proposal starts from a simple observation about human communication: people often say more than they need to ("that red mug", with a point) or point more clearly than they speak. So in unlabeled video, the modality that happens to be clear can label the one that is ambiguous. We propose to train two target resolvers together on unlabeled situated video. One reads pointing, body pose and gaze. The other reads the spoken words and the conversation so far. Each is trained toward the combined judgment of both on clips where that combination is confident. A small labeled set is then enough to calibrate the pair.

The question is whether this "symmetric" supervision lets a deictic resolver match fully supervised accuracy with a small fraction of the labels. The rival would be generic self-supervised pretraining, such as masked reconstruction of video and audio. The idea is new here because it turns a decades-old account of how gesture and language make up for each other (Natural and Simulated Pointing: gestures let speakers shorten descriptions, words narrow vague points) into a training signal. Existing label-efficient work only reconstructs co-occurring modalities or uses gaze as a silent filter.

Core Claim

Deictic reference resolution means picking the physical referent of words like this, that or it. In natural data the two cues often carry the same information, but they fail in different ways. A point toward a dense cluster is vague but spatially reliable. The words "the red one" are precise about attributes but say nothing about where the object is. When the two cues' errors are roughly independent given the true referent, the textbook condition for co-training (training two models on separate views of the same data, each labeling examples for the other), agreement between them is a cheap and fairly clean stand-in for a human annotation.

We claim that this signal is more useful for referent selection than reconstruction-style pretraining. Reconstruction teaches which pixels and words co-occur. Cross-modal agreement teaches which candidate object is the referent. We expect the benefit to be concentrated exactly where one modality alone is ambiguous. That prediction can be checked, and it separates our mechanism from a generic gain from more data.

What Prior Work Establishes and Leaves Open

The strongest resolver is TransGesture from Toward Human Deictic Gesture Target Estimation. It pairs a frozen DINOv2 backbone with gaze and gesture decoders, fused by joint cross-attention. It reaches 89.01% existence accuracy and 58.48% mask IoU on its GestureTarget benchmark of more than 20,000 instances, where humans reach 90% and 80%. That paper supplies our gesture branch and several constraints. Gaze adds about 6 IoU points. DINOv2 features keep the spatial link between finger and target much better than CLIP or SigLIP features. GPT-4o scores below 5% mask IoU. But TransGesture has no speech input, trains its decoders on all the labels, and reports no low-label results.

On the language side, DeiSAM: Segment Anything with Deictic Prompting resolves relational text prompts without task labels by chaining a scene-graph generator, LLM-written logic rules and SAM. It reaches 87.83% mAP on DeiVG3 against 75.79% for LISA. It has no pointing or audio. It also documents noun confounding in neural grounders: they segment the object named in the phrase rather than the intended referent. Our language branch has to be tested against this failure. Older work motivates keeping uncertainty rather than committing early. Joint interpretation of input speech and pen gestures aligns speech and pen gestures over hypothesis lattices so that each modality prunes the other's readings. Gesture Improves Coreference Resolution shows that gesture features measurably help resolve spoken coreference. Both use supervision or hand-built features. We use the same cross-modal constraint to generate training signal.

On the label-efficiency side, three results show that human attention and paired modalities can stand in for labels. Ultrasound Image Representation Learning by Modeling Sonographer Visual Attention pretrains on sonographer gaze and matches a supervised network with 30 times fewer labels. Active Gaze Behavior Boosts Self-Supervised Object Learning shows that toddler gaze curates good contrastive training views. Advancing Radiograph Representation Learning with Masked Record Modeling jointly masks images and reports and reaches 88.5% mean AUC on CheXpert with 1% of the labels. All three treat attention or text as a pretext for general representations. None uses one communicative channel to decide the referent for another. The masked-modeling recipe is the most natural competitor, and we adapt it as a baseline.

Finally, the situated benchmarks set the stakes. Can Vision-Language Models Answer Face to Face Questions in the Real-World? (QIVD) finds that adding audio to pretrained vision models without situated fine-tuning can hurt. We read this as evidence that naive fusion fails, not that fusion is useless. "Is This It?" reports "it" in 21% of situated utterances versus 3% in OpenEQA, and 42% self-talk. Many real references therefore carry almost no descriptive content, which caps how often speech can teach gesture. We treat this as a limit to measure.

Research Program

Both resolvers score the same set of candidate regions. The candidates are class-agnostic SAM masks over frozen DINOv2 features, so the backbone is shared and fixed as in TransGesture. The gesture resolver $G$ takes the gesturer's body mask, hand keypoints and gaze estimate and outputs a distribution $p_G$ over candidates. It uses TransGesture's decoder design. The speech resolver $L$ takes a transcript window around the deictic word plus the preceding dialogue. It scores candidates by region–text similarity plus a small discourse-recency term, and outputs $p_L$.

On an unlabeled clip we form the combined estimate $q \propto p_G \, p_L$. This is a product of the two distributions, which is the right way to combine them if their errors are independent. We then train each resolver toward it:

$$ \mathcal{L}_{\text{unlab}} = w(q)\,\big[\mathrm{KL}(\bar q \,\|\, p_G) + \mathrm{KL}(\bar q \,\|\, p_L)\big] $$

Here $\bar q$ is $q$ with gradients stopped, so each resolver is pulled toward the joint judgment but cannot move that target by changing itself. The weight $w(q)$ is one minus the normalized entropy of $q$, so only clips where the combination is confident contribute. The small labeled set adds ordinary cross-entropy to both resolvers. We keep $q$ soft rather than thresholding it, following the lattice-fusion argument.

The design stands or falls on the independence assumption, so we measure it directly. On a labeled development split we compute how often $G$ and $L$ make the same wrong choice. We report this before claiming any gain. If shared errors are common, for example because both are drawn to the most visually salient object, the method should fail, and we will say so.

The unlabeled pool comes from the source video of GestureTarget's constituent corpora that carry dialogue (Werewolf Among Us, Social-IQ, SocialGesture), with transcripts from Whisper, plus unlabeled QIVD clips. One caveat: GestureTarget instances are derived frames, and we have not verified how many can be aligned to their source utterances. Recovering that alignment is the first engineering task. The speech-bearing subset becomes our main evaluation split, and results on the full benchmark are reported with a gesture-only $G$ at test time.

Baselines

The naive reference point is geometric ray casting. It extends the vector from elbow through fingertip and picks the first candidate mask the ray hits, using no learning and no labels. It sets the floor that any learned resolver must clear and shows how often pointing alone is unambiguous.

The strongest existing method is TransGesture (Toward Human Deictic Gesture Target Estimation), reported at 89.01% existence accuracy and 58.48% mask IoU on GestureTarget with full labels. No reported result was found on Lacuna for TransGesture at 1% or 10% of labels. We retrain it ourselves at 1%, 5%, 10% and 100% label fractions on the same splits, with the same frozen DINOv2 and the same hyperparameter search budget as our method. This is the competitor most likely to embarrass us, because its backbone is already self-supervised and its decoders may need few labels. Beating it means higher mask IoU at 10% labels with non-overlapping seed intervals, and ideally matching its 100%-label IoU at 10% or less.

The second learned competitor is masked cross-modal pretraining in the style of Masked Record Modeling (Advancing Radiograph Representation Learning with Masked Record Modeling). We mask patches and transcript tokens in the same unlabeled pool, pretrain the same decoders on reconstruction, and then fine-tune on the same label fractions. It receives equal unlabeled data and equal pretraining GPU-hours. It is the fairest test of whether our referent-level signal beats a representation-level one.

Two ablations isolate the key ingredient. The first is single-model self-training: $G$ trains on its own confident predictions, with no speech. It tests whether the gain comes from pseudo-labeling in general rather than from the second modality. The second is one-directional transfer, where speech teaches gesture but not the reverse. It tests whether the symmetry matters. We also include GPT-4o's reported below-5% mask IoU from the same paper as a zero-shot reference, without rerunning it.

Why not just…?

Why not just fine-tune TransGesture on the small labeled set? This is the real threat. DINOv2 already provides spatially coherent self-supervised features, and a light decoder might learn target selection from 10% of the labels with little loss. Nothing on Lacuna reports its low-label curve, so our label-fraction sweep settles this directly. If its 10% IoU lands within noise of ours, the co-training contribution is not worth its complexity, and we will report that.

Why not just pretrain with masked reconstruction on unlabeled speech-and-video? Masked Record Modeling shows that paired-modality reconstruction buys large label savings in radiology. But reconstruction rewards predicting what co-occurs, and a pointed-at object and a merely nearby one co-occur equally well. The matched-compute comparison above tests this. Our specific prediction is that the reconstruction baseline will gain uniformly, while our method will gain most on clips where the ray-casting baseline is ambiguous.

Why not just use a modular pipeline or prompt a large multimodal model? GPT-4o scores below 5% mask IoU on gesture targets (Toward Human Deictic Gesture Target Estimation). Text-only grounders show noun confounding (DeiSAM). Naively added audio degraded modular systems in QIVD (Can Vision-Language Models Answer Face to Face Questions in the Real-World?). A modular pipeline is nonetheless what our $L$ branch resembles at initialization. Reporting $L$ alone and $G$ alone before co-training quantifies exactly what joint training adds over stitching the pieces together.

Proposed Experiments and Decisive Checks

The primary experiment is the label-fraction sweep on the speech-aligned GestureTarget split. We report mask IoU and existence accuracy for ray casting, TransGesture, the masked-pretraining baseline, both ablations and our method, over three seeds. The mechanism check stratifies test clips by ambiguity: how many candidates the pointing ray intersects, and whether the utterance contains a descriptive attribute or only a bare pronoun. Our account predicts that gains grow with ambiguity for utterances that carry descriptions, and shrink toward zero for bare "it". The error-correlation measurement comes before either result and tells us whether the co-training assumption holds.

For transfer, we evaluate the deictic and object-referencing categories of QIVD. We found no per-category number on Lacuna, only GPT-4o's 58.76% overall accuracy. We therefore run the baselines ourselves on those categories. We also run a qualitative error analysis on the Sigma corpus, whose 284 utterances are too few for reliable statistics, to see how often ecologically valid references fall in the bare-pronoun regime where speech cannot help.

The proposal succeeds if three things hold together: co-training beats low-label TransGesture and matched-compute masked pretraining at 10% labels, the gains concentrate where one modality is ambiguous, and measured cross-modal error correlation is low. It fails informatively if fine-tuned TransGesture already closes the gap, which would show that the bottleneck in label-efficient deictic resolution lies elsewhere than in target supervision.