In written language, we have spaces. In spoken language, we have pauses. But in sign language, the "words" (known as glosses) flow into one another in a seamless, continuous stream. This fluidity creates a fundamental problem for computer vision: if you don't know where one sign ends and the next begins, how do you translate the sentence?
Researchers in Continuous Sign Language Recognition (CSLR) are moving away from identifying isolated gestures toward systems that can parse this "fluid stream" by modeling the complex temporal relationships between movements.
The Problem of Co-articulation
The primary hurdle in CSLR is a phenomenon called co-articulation. As Oishi Deb, Andrew Zisserman, and their colleagues explain in New keypoint-based approach for recognising British Sign Language (BSL) from sequences, signs blend together because the hands are already moving toward the position of the next sign before the current one is finished.
This means a sign doesn't have a fixed "look." Its beginning is influenced by the previous sign, and its ending is pulled toward the next. Lionel Pigou and his team noted in Gesture and Sign Language Recognition with Temporal Residual Networks that this lack of clear temporal boundaries makes traditional segmentation—cutting a video into discrete chunks—nearly impossible and highly error-prone.
Hierarchical Features and Semantic Pyramids
To solve this, researchers have developed architectures that look at video at multiple time scales simultaneously. The intuition is that while a single frame might be ambiguous, the "rhythm" of a sign emerges over several frames, and the "meaning" of a sentence emerges over several signs.
Dongxu Li and Xin Yu proposed a "Temporal Semantic Pyramid" in their paper TSPNet. By evaluating video segments at different scales, the model can capture fine-grained finger movements at one level and broader arm trajectories at another. Similarly, Zhaoyang Yang introduced SF-Net, which structures features across three levels: the frame, the gloss, and the sentence. This hierarchy allows the model to learn the internal structure of a sign while maintaining the context of the entire message.
Recognition Without Explicit Segmentation
If finding boundaries is the problem, some researchers suggest skipping that step entirely. Jie Huang and Wengang Zhou pioneered Video-Based Sign Language Recognition Without Temporal Segmentation, using architectures that map the entire visual sequence directly to a sequence of words.
However, recent work suggests that a "soft" understanding of boundaries still helps. In MHB: Multimodal Handshape-aware Boundary Detection, Mingyu Zhao and Dimitris N. Metaxas use a specialized handshape classifier to help the AI "feel" the transitions between signs. By focusing on when the hand configuration changes, the system can more accurately anchor the start and end points of a gesture within the continuous flow.
Focusing on the Skeleton
A recurring theme in the most recent papers is the shift from raw pixels to skeletal keypoints. Processing every pixel in a video is computationally expensive and introduces noise from backgrounds or lighting.
Qi Guo and Shujun Zhang argue in their work on Spatial-Temporal Graph Attention Networks that the "signal" of sign language lives in the joints of the hands and body. By treating the human skeleton as a graph that evolves over time, models can ignore the "scenery" and focus entirely on the geometry of the motion. This approach, also championed by the VGG team in their BSL recognition work, allows for much larger vocabularies—scaling up to thousands of signs—because the model is learning the fundamental "physics" of the language rather than just memorizing pixel patterns.