Abstract
Integrating continuous human movement data into discrete Large Language Model (LLM) architectures is a significant challenge. This paper presents a method for tokenizing American Sign Language (ASL) fingerspelling into discrete units using Residual Vector Quantization (RVQ) and Connectionist Temporal Classification (CTC) loss. By transforming continuous hand-pose coordinates into "character-aware" tokens, the system achieves high interpretability and performance on downstream translation tasks, outperforming continuous baselines in several metrics.
Introduction
Modern Large Language Models (LLMs) excel at processing discrete tokens like text. However, signals such as audio, video, and human gestures are inherently continuous. In the domain of Sign Language, models typically rely on raw video or continuous embeddings, which are computationally intensive and lack interpretability. This research addresses the problem of discretization: converting the fluid motion of ASL fingerspelling into a sequence of discrete, meaningful tokens that preserve both semantic meaning and visual nuance.
Methodology
The proposed system treats ASL pose data as a sequence of frames, similar to neural audio codecs.
Pose Representation and Architecture
To ensure privacy and efficiency, the model uses (x, y) coordinates for 42 hand landmarks extracted via MediaPipe instead of full video frames. The core architecture is a modified Squeezeformer, which combines the local feature extraction of Convolutional Neural Networks (CNNs) with the global context modeling of Transformers.
Tokenization via Residual Vector Quantization (RVQ)
The "tokenizer" employs RVQ to pass the signal through multiple codebook layers. This creates a hierarchical representation where the first layer captures broad hand shapes and subsequent layers capture fine-grained joint details.
Character-Awareness with CTC Loss
A critical innovation is the addition of Connectionist Temporal Classification (CTC) loss during training. While standard autoencoders focus on reconstruction, CTC loss forces the model to align video frames with ground-truth characters. This ensures that the resulting tokens are semantically grounded rather than just visually clustered.
Experiments and Results
The model was trained on the ASL Fingerspelling dataset, containing over 3 million characters from 100+ signers.
Probing and Interpretability
Probing tasks revealed that discrete tokens with CTC loss achieved accuracies between 90.8% and 99.0% in identifying specific characters. Furthermore, the inclusion of CTC loss significantly boosted the Mutual Information score from 0.090 to 0.455, indicating a much stronger link between the tokens and the underlying characters.
Translation Performance
In sequence-to-sequence translation (sign tokens to text), the discrete model achieved a normalized Levenshtein distance of 0.690, slightly outperforming the continuous baseline (0.684). This suggests that the "information bottleneck" of discretization helps filter out noise and focus on salient features.
Discussion and Limitations
While successful for fingerspelling, the current approach does not yet encompass the full complexity of ASL, which includes facial expressions and body posture. The system is also dependent on the accuracy of the initial landmark detection. Future work will explore scaling this hierarchical tokenization to full-body ASL and generative applications.
Summary updated