In spoken languages, we rely on textual pronouns to keep track of discourse entities. If we say, "Alice met Bob, and she gave him a book," the words she and him resolve to Alice and Bob through sequential context.
In signed languages, this tracking mechanism is geometric rather than lexical. As Yin et al. explain, signers utilize the physical three-dimensional space around them as a persistent functional database. To introduce an entity into a conversation, a signer assigns it to a specific spatial coordinate in the air, known as a locus. Later, instead of using a spoken pronoun, the signer simply points back to that exact coordinate (spatial indexing) or inflects a verb's movement between coordinates to denote subjects and objects (spatial agreement).
Alikhani notes that this visuospatial entity tracking provides a unique framework for studying language grounding: it transforms abstract discourse entities into physical coordinates within the visual field. For a computational model to understand or translate natural sign language, it cannot treat the video as a simple sequence of independent gestures; it must maintain an internal, persistent map of this 3D signing space.
The Geometry of Grammar
To understand why spatial tracking is necessary, consider how grammatical relationships are constructed. In a linear text language like Japanese, grammatical roles are determined by word order or particles. However, when translating text into Japanese Sign Language (JSL), Takahashi demonstrates that these relationships must be mapped onto physical space. A translation system must assign 3D coordinates to the subject and object, and then alter the physical trajectory of the verb sign so that it moves from the subject's locus to the object's locus.
Because the 3D space acts as a memory buffer, the meaning of a sign is entirely dependent on its spatial context. If a model observes a pointing gesture but has no record of what was previously anchored to that coordinate, the gesture is linguistically empty.
Why Sentence-Level Models Break
Historically, computer vision and natural language processing models have treated sign language translation as a sentence-level video-to-text problem, chopping continuous signing videos into short, isolated clips. Recent research reveals that this framing is fundamentally flawed.
Tanzer et al. discovered that roughly one-third of isolated sign language clips are completely untranslatable—even by fluent human signers—because the spatial referents lose their antecedents. If a clip begins after a signer has established that the left side of the air represents "the manager" and the right side represents "the customer," a subsequent pointing gesture to the left cannot be resolved in isolation.
This issue is exacerbated by standard dataset design. As Costa-jussà et al. point out, datasets consisting of isolated sentence recordings omit the natural flow of continuous discourse, preventing models from learning how spatial placements are established and maintained over time. When models are trained on these fragmented clips, they fail to generalize to real-world communication where context spans multiple sentences, a limitation documented across various sign languages by Joshi et al. and Jang et al..
Sentence-Level Modeling (Broken):
[Clip A: "The manager met..."] -> [Clip B: "He pointed (Left)"] -> Who is "Left"? (Context Lost)
Discourse-Level Modeling (Correct):
[Continuous Video] -> Maintain 3D Coordinate Map: {Left: "Manager", Right: "Customer"} -> Resolve Pointing Gesture
Engineering a Persistent Spatial Memory
To solve this problem, computational architectures must shift from processing static frames or short clips to performing Signed Coreference Resolution (SCR).
The first computational attempt to formalize this task was introduced by DeHaan et al.. Rather than tracking text tokens, the authors designed an unsupervised multigraph model that tracks spatial and temporal heuristics across a 3D coordinate map. By maintaining a persistent representation of where entities are anchored in the signing space, the system can track references over long-form discourse, linking subsequent pointing gestures back to their original noun antecedents.
Scaling this to unconstrained, real-world data remains an open challenge. Benchmarks like Auslan-Daily, introduced by Shen et al., show that standard deep learning architectures still lack the explicit spatial reasoning and long-term memory required to track these visual pronouns in complex, multi-person environments. Bridging this gap requires models that treat the 3D visual field not merely as a collection of pixels, but as a continuous, structured database of discourse context.
Key Foundational Work
Including Signed Languages in Natural Language Processing (2021) — contributed 3 concept elements to this direction.
An Approach to Machine Translation from Japanese Text to JSL Text (2008) — contributed 2 concept elements to this direction.
Signed Coreference Resolution (2021) — contributed 1 concept elements to this direction.