Current models of two-person motion let the reacting person "see" everything the other person does, including movements behind their back. When one person taps another on the shoulder from behind, the real reaction starts when the tap is felt or the hand comes into view, not when the arm first starts to move. A model that reads the toucher's full trajectory can begin turning early, and reported metrics will not catch it. We propose to give a discrete motion model an explicit, time-varying record of what the reacting person knows about the other's body. That record is a knowledge fluent: a property that switches on when something is sensed, persists afterward, and grows stale. The reaction generator then conditions on this belief record instead of the partner's true motion.
The idea pairs two literatures that have not met. Motion tokenizers compress continuous body movement into a vocabulary of integer codes so that a transformer can model it the way it models text. Logics of knowledge over time treat "agent $i$ knows $f$ at time $t$" as a fluent that obeys rules for sensing, persistence, and loss. Our claim is that real human reaction data already carries the signature of partial observability. A model that encodes it explicitly should predict real reactions more accurately, especially their timing, and should not leak information the reactor could not have had. The claim can be tested on existing datasets using head orientation that is already recorded, so it does not rely on synthetic false-belief scenarios that have no ground truth.
Core Claim
Take a masked token model of two-person interaction. Replace the reactor's direct view of the actor's tokens with a per-body-part knowledge track: what the reactor last perceived of each actor body part, how it was perceived (sight or contact), and how long ago. This track should do three things. First, it should lower prediction error on reactor motion in segments where the actor is partly out of the reactor's view. Second, it should move predicted reaction onsets toward the true perceptual onset. Third, it should remove counterfactual leakage, meaning the reactor's output changes when unseen actor motion is swapped out. It should do this without hurting overall fidelity.
We also make a sharper claim and treat it as the main risk. The explicit fluent tokens, with persistence and staleness, should beat a simpler baseline that only hides invisible frames from attention. If they do not, the honest conclusion is that perceptual masking is enough and the fluent machinery is not needed.
Grounding in Prior Work
The motion side supplies the method. InterMask tokenizes two-person motion into a grid of time by body part using a vector-quantized autoencoder, which learns a codebook of discrete motion snippets. It then predicts masked tokens with cross-attention between the two actors, and it can generate one person's reaction zero-shot by freezing the other's tokens. We reuse this body-part grid because it lets knowledge be tracked per limb. InterMask's cross-attention, like the symmetric mutual attention in InterGen and the hierarchical conditioning in HINT, assumes each person observes the other completely at every frame. That assumption is exactly what we remove.
BEST motivates working in discrete space. It discretizes coupled hand and body poses and shows that classifying tokens beats regressing continuous coordinates for recognition, at 58.82% versus 49.54% top-1 pose-only on MSASL1000.
The knowledge side supplies the update rules. Reasoning with Knowledge, Action and Time in Dynamic and Uncertain Domains formalizes knowledge as a linear-time fluent $Knows(f,t)$. It has three relevant ingredients: inertia, so knowledge persists once acquired; release, so knowledge is lost after unobserved non-deterministic change; and indirect sensing through causal dependencies. Indirect and Conditional Sensing in the Event Calculus shows how unobserved causes can be inferred from observed effects. Both works operate purely over symbolic propositions. We compile their update rules into a deterministic preprocessor that runs over geometry.
Human-Aware Epistemic Task Planning and Epistemic Plan Recognition motivate the problem. They show that belief divergence from unshared experience matters for collaboration. They also show that symbolic tracking blows up combinatorially: more than 5,600 states for three cubes in the human-aware planner. Recognition also degrades badly when the observer's belief model is wrong. We avoid enumerating possible worlds. The tracked belief is a single "last-perceived" content plus staleness, and the learned model handles the uncertainty that follows.
GIMO shows that egocentric gaze improves motion prediction, reducing destination translation error from 0.393 to 0.245. That supports perception as a predictor of movement. GIMO, however, feeds continuous gaze in as a feature and does not represent knowledge or its persistence.
Research Program
The first component is a perception model. For each frame of InterHuman and Inter-X, we compute the reactor's view cone from head orientation, using the head joint in InterHuman's SMPL-H skeleton and the head and face in Inter-X's SMPL-X. We cast rays to decide which actor body parts are visible, accounting for self-occlusion by both bodies. A separate contact detector, based on surface proximity between meshes, marks parts sensed by touch.
Head orientation is only a proxy for gaze. We calibrate it on GIMO, which records true eye gaze: we measure how often the head-based visibility label disagrees with the gaze-based one, and we use a widened cone as a sensitivity setting.
The second component is the knowledge fluent track, built with event-calculus update rules. For reactor $B$, actor body part $p$, and frame $t$, the fluent state $s_{B,p,t}$ takes one of a few discrete values: currently seen, currently felt, remembered with staleness falling into one of $k$ logarithmic bins, or never perceived. Sensing initiates knowledge. Inertia carries it forward and increments staleness. Release resets the part to "unknown-but-moving" once staleness passes a horizon. This mirrors how the linear-time knowledge theory handles knowledge loss under unobserved change.
Alongside the state, the track stores a content token $c_{B,p,t}$. This is the actor's motion token for part $p$ at the most recent frame when $B$ perceived it. Both the state and the content are integers, so they slot into the existing token grid with no new continuous machinery.
The third component is the belief-conditioned reactor. An InterMask-style masked transformer generates the reactor's tokens while cross-attending only to the knowledge track $(s, c)$ for the actor, never to the actor's true tokens. The training objective is unchanged: masked-token cross-entropy on the reactor's real motion. If real humans react to what they perceived, the belief-conditioned model should fit held-out reactions better in partially observed segments, even though it sees strictly less information.
A by-product is that the track gives a readable symbolic belief state for each frame and body part, of the kind that symbolic epistemic planners consume. This is a step toward the continuous-to-symbolic grounding gap. We report it only as a qualitative interface.
Baselines
The naive reference point is InterMask's released zero-shot reaction setting: freeze the actor's true tokens and generate the reactor with full cross-attention. This is the "omniscient reactor" and represents the status quo assumption. We retrain it with our codebook and data splits so that the only difference from our model is what the cross-attention reads.
The strongest existing method on overall interaction fidelity in the file is InterMask itself. It reports FID 0.399 on Inter-X and 5.154 on InterHuman, the latter exceeded in the file only by HINT's 3.100. For reaction generation on Inter-X, the file reports a reaction FID of 4.386 for AGRoL. There is no reported result on Lacuna for InterMask's own reaction-only metrics, so we measure them ourselves.
We also include HINT as a second architecture, retrained with the same omniscient conditioning. This checks that any gain is not specific to InterMask. Because HINT's canonicalized local frames separate each person's motion from the relative transform between them, it is a natural place to test whether hiding information interacts with coordinate choice.
The key ablation is history-masked attention. The reactor attends to the actor's true tokens only at frames and parts that were visible or in contact, and it has access to all past visible frames. This gives persistence implicitly through attention, with no fluent tokens. It isolates our key ingredient: explicit staleness and sensing-mode tokens, together with release.
A weaker ablation is instantaneous masking, which shows only currently visible parts and no history. It tests whether memory matters at all. A final control shuffles visibility labels across sequences while keeping their marginal statistics. This tests whether gains come from perception specifically, rather than from adding any information bottleneck.
All variants share one tokenizer, identical data splits, and the same parameter count within 2%. Each gets the same hyperparameter search, a fixed number of configurations chosen on validation reactor likelihood. Each is trained under the same compute budget with three seeds.
Beating the omniscient baseline means three things. Reactor likelihood must be lower on the partially observed subset. Onset error must be smaller by more than the seed spread. Overall reaction FID must not get worse by more than the seed spread.
Why not just…?
Why not just keep full cross-attention and let the model learn to ignore what the reactor cannot see? Nothing in the training objective penalizes using unseen information. When a full-view model has access to the actor's early motion, using it lowers training loss on average. It will therefore learn to anticipate from information that is physically unavailable. The counterfactual leakage test below measures this directly. If the omniscient model shows near-zero leakage and matches our onset error, this alternative wins and the paper's claim fails.
Why not just mask invisible frames from attention? This is the real threat, and we treat it as the main ablation rather than a strawman. Attention over all past visible frames already supplies "remembered" content. Our explicit track adds three things that masking lacks: a sense of how stale a memory is, independent of position, the distinction between sight and touch, and release after long occlusion. These matter only if human reactions depend on them, for example if people hedge more as memories age. If history-masked attention matches us on onset error and likelihood, we will report that perceptual masking captures the effect. The contribution then narrows to the perception model and the evaluation protocol.
Why not just concatenate continuous gaze or view vectors as extra input, as GIMO does? A continuous visibility feature tells the model what is visible but still hands it the actor's full motion. Leakage therefore remains possible. Concatenation also loses the persistence structure. We run this variant: visibility flags concatenated to omniscient inputs. It should help onset timing somewhat, consistent with GIMO's gains, while still failing the leakage test.
Why not run a symbolic epistemic planner over classified actions? The planner and plan-recognition results show possible-world blow-up and brittleness when the observer's model is wrong. A classify-then-plan pipeline also produces canned motion, so it cannot be scored on held-out reactor likelihood. We do not consider it competitive on this question and do not run it.
Proposed Experiments
The decisive check is a perceptual-onset test on naturally occurring data. Using Inter-X's actor and reactor labels, we select sequences where the actor's initiating body parts are outside the reactor's head-based view cone when the action begins. Examples include approaches from behind and shoulder taps. In each, we mark the true perceptual onset: first visibility or first contact. We also mark the true reaction onset, the first frame where reactor joint velocity exceeds a threshold.
Each model generates reactor motion from the actor's motion. We measure two quantities. The first is the absolute error between predicted and true reaction onset. The second is whether the predicted onset falls before perceptual onset, which no human reactor can do.
A second check is counterfactual leakage. On all sequences, we replace the actor's motion during reactor-invisible windows with motion drawn from another sequence, keeping the visible frames fixed. We then measure the change in the generated reactor motion within those windows. The belief-conditioned model should be invariant up to sampling noise; the omniscient model should not be.
Standard measures complete the picture. These are reactor masked-token likelihood and joint position error on the partially observed subset versus the fully visible subset, plus reaction FID and interaction FID on both InterHuman and Inter-X. Together they confirm that fidelity is preserved where the actor is fully visible.
Robustness runs repeat everything with the widened view cone calibrated on GIMO and with contact sensing removed. We also report how many sequences fall into the partially observed subset. If that subset is too small to support conclusions, we will add synthetic occluders as a stress test and label that analysis as synthetic. A further risk is that existing capture data, recorded in open studios, contains little true occlusion. That would cap effect sizes, and the counts will show it before any training is done.
References
- InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling
- InterGen: Diffusion-Based Multi-human Motion Generation Under Complex Interactions
- HINT: Hierarchical Interaction Modeling for Autoregressive Multi-Human Motion Generation
- BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization
- Reasoning with Knowledge, Action and Time in Dynamic and Uncertain Domains
- Indirect and Conditional Sensing in the Event Calculus
- Human-Aware Epistemic Task Planning for Human-Robot Collaboration
- Epistemic Plan Recognition
- GIMO: Gaze-Informed Human Motion Prediction in Context