When humans learn a language, we do not memorize every word as an indivisible, unique acoustic waveform. Instead, we intuitively understand that words are built from a finite alphabet of reusable sounds—phonemes.
Yet, for years, standard computer vision models treated human gestures and sign language as monolithic entities. To recognize a sign like "house," a model would attempt to map the entire continuous video sequence directly to a single semantic label. This holistic approach creates a severe bottleneck: it requires massive amounts of training data for every individual sign, struggles to generalize to unseen vocabularies, and fails entirely on low-resource sign languages.
To overcome this, a powerful paradigm shift borrows directly from linguistics: sub-lexical decomposition. By breaking continuous, complex human movements down into a finite inventory of discrete, reusable structural building blocks—analogous to phonemes or letters—models can learn the underlying "grammar" of motion.
The Anatomy of a Gesture Phoneme
What does a phoneme look like when it isn't spoken, but seen? In articulatory speech recognition, researchers long ago realized that speech could be modeled as a sequence of physical gestures tied to specific vocal organs, such as tongue movements or lip closures, rather than raw acoustic frequencies (Gutkin et al.).
Applying this logic to human body movement reveals that complex gestures are actually combinations of distinct, parallel physical attributes. In sign language, these are known as phonological features:
- Handshape: The specific configuration of the fingers (e.g., extended, flexed, thumb position).
- Location: Where the sign occurs relative to the body (e.g., near the forehead, chest, or chin).
- Movement: The trajectory or type of motion (e.g., circular, straight, oscillating).
- Orientation: The direction the palm or fingers are facing.
Holistic Approach:
[ Video of Sign ] ───────────────────────────────> "Apple" (Requires massive data per sign)
Sub-Lexical Approach:
┌──> Handshape: Flat Palm
[ Video of Sign ] ──┼──> Location: Cheek ──> Combined to recognize or generate "Apple"
└──> Movement: Twist
Instead of forcing a neural network to map raw pixels directly to a concept, sub-lexical decomposition introduces an intermediate representation layer. For instance, Kim et al. demonstrated that abstracting raw visual data into intermediate phonological features like finger extensions or hand orientations makes fingerspelling recognition significantly more robust. Because these linguistic building blocks are invariant to individual signer styles and clothing, the model filters out environmental noise and focuses purely on semantic structure.
Similarly, Paudyal et al. leveraged this modularity to build explainable AI for sign language learners. Because their system evaluates location, movement, and handshape independently, it does not just output a binary "correct/incorrect" classification; it can pinpoint exactly why a sign was wrong (e.g., "correct handshape, but wrong location").
Why Compositionality Scales Exponentially
The primary reason sub-lexical decomposition works so well is combinatorial scaling. If a model must learn 1,000 signs holistically, it needs thousands of examples for all 1,000 signs. But if those 1,000 signs are composed of just 10 fundamental handshapes and 5 locations, the model only needs to master those 15 primitives.
The authors of Talking With Your Hands exploited this property to scale gesture recognition exponentially. By training a convolutional neural network to recognize just 10 fundamental hand shapes ("gesture-phonemes"), their system successfully recognized over 800 complex gesture sequences. The training data requirements scaled linearly with the number of primitives, while the recognizable vocabulary scaled exponentially.
This compositionality is crucial for handling low-resource languages and out-of-vocabulary instances. As Tavella et al. showed with the WLASL-LEX dataset, graph-based neural networks trained on phonological properties can successfully recognize components of signs the model has never encountered during training. If a model encounters an unfamiliar sign, it can still decode its constituent parts—such as a specific hand arrangement or spatial placement (Cooper et al.).
To prevent deep learning models from accidentally collapsing these distinct features back into a single holistic shortcut, modern architectures enforce structural boundaries. For example, Cheng et al. introduced an orthogonality loss in their state-space model framework, PHONSSM. This loss mathematically forces the latent representations of handshape, location, and movement to remain orthogonal, ensuring that the model treats them as truly disentangled, independent factors.
Syntactic Structures and Motion Codebooks
The analogy between human motion and natural language extends beyond recognition into motion synthesis and generation. If primitive gestures act as "words," their temporal ordering can be governed by a "grammar" (Wang et al.). By learning context-free grammars directly from motion capture data, systems can automatically discover hierarchical syntactic structures within human movement, allowing for the generation of realistic, novel actions (Park et al.).
In Sign Language Production (SLP), this is often operationalized through a learned motion codebook. Instead of predicting continuous, high-dimensional 3D joint coordinates frame-by-frame—which often leads to blurry or unrealistic animations—Walsh et al. discretized continuous pose data into a sequence of tokens from a compact codebook. When the size of this motion codebook is significantly smaller than the total vocabulary of the language, it serves as empirical proof that the system has successfully discovered reusable sub-units of motion.
These discrete primitives can then be dynamically recombined. By utilizing a "Mixture of Motion Primitives" (MOMP), Saunders et al. demonstrated that a model can blend specialized motion "experts" at the frame level, synthesizing an infinite variety of continuous signs from a small, compact set of base movements.
Conclusion
Whether analyzing daily egocentric human activities via hierarchical language annotations (Ma et al.) or parsing isolated sign language videos into "Visual Symbols" (Li et al.), treating human motion as a structured, phonological entity fundamentally changes how machine learning models process physical actions. Moving from whole-sign recognition to the identification of sub-lexical visual units (Cooper et al.) provides the inductive bias necessary to make gesture and sign language models scalable, explainable, and robust (Kezar et al.).
Key Foundational Work
Unsupervised Analysis of Human Gestures (2001) — contributed 2 concept elements to this direction.
Structural Representation and Matching of Articulatory Speech Structures based on the Evolving Transformation System (ETS) Formalism (2005) — contributed 1 concept elements to this direction.