TL;DR
Researchers have introduced CNSL-bench, the first comprehensive benchmark designed to evaluate how well Multimodal Large Language Models (MLLMs) actually understand Chinese National Sign Language. The results are a wake-up call: even top-tier models like GPT-5 struggle significantly with video-based sign language, trailing human performance by over 40 percentage points.
What Problem Does This Paper Address?
While Large Language Models (LLMs) have revolutionized natural language processing, sign language remains a "final frontier" for AI. Sign language isn't just a series of simple gestures; it is a complex, linguistically grounded system involving hand shapes, motion trajectories, and facial expressions.
Until now, most research has used LLMs as "black box" decoders at the end of a translation pipeline. We haven't really known if MLLMs possess the intrinsic ability to perceive and interpret the linguistic structure of signs directly from visual inputs. Furthermore, sign language varies by region, and the lack of a standardized, multimodal benchmark has made it impossible to compare different models fairly. CNSL-bench fills this gap by providing a massive, standardized dataset anchored to the official Chinese National Sign Language dictionary.
Methods
The authors constructed CNSL-bench using three core principles:
- Authoritative Grounding: They used the National Common Sign Language Dictionary to ensure every sign has a canonical, non-ambiguous definition.
- Multimodal Alignment: Every sign entry is represented in three ways: a textual description, an illustrative drawing, and a real-world video.
- Articulatory Diversity: The benchmark specifically categorizes signs into three linguistically distinct forms: air-writing (tracing characters in the air), finger-spelling (depicting character structures), and the manual-alphabet (Pinyin-based finger configurations).
To evaluate the models, the researchers used a four-way multiple-choice format. This "closed-form" design allows for scalable, objective testing, as current models are still too unreliable for open-ended sign language generation. They also employed Test-Time Scaling (Chain-of-Thought reasoning) to see if giving models more "time to think" would bridge the performance gap.
Datasets
- CNSL-bench (New): The primary contribution, featuring 6,707 unique sign entries and 20,121 evaluation instances across text, image, and video modalities.
- CNSL-DP: A large-scale video dataset used to provide the synchronized video recordings for the benchmark.
- CSL-Daily: Used for a qualitative case study to show how flagship models fail at open-ended sentence-level sign understanding.
Key Insights and Ideas
- The Modality Gap: Models are "text-heavy." They perform reasonably well when reading a description of a sign but fall apart when watching a video of the same sign. This suggests that visual-temporal grounding is the primary bottleneck.
- Discrete vs. Continuous: Models are much better at finger-spelling (which is more discrete and character-like) than air-writing or manual-alphabets, which involve continuous, shape-intensive motions.
- Reasoning Isn't a Silver Bullet: While "slow thinking" (Chain-of-Thought) helped models like Gemini-2.5-Flash, it actually led to negative gains for others like Qwen3-VL-Plus on certain tasks. Top-tier models also hit a "boundary effect" where more reasoning tokens eventually degraded performance in video settings.
- Hallucination in Motion: Case studies revealed that when models fail at video, they don't just get it wrong—they hallucinate entirely different gestures (e.g., seeing a "typing" motion but interpreting it as a "crossing" gesture).
Key Results
The study evaluated 21 state-of-the-art MLLMs. The quantitative findings highlight a massive disparity:
- Human Baseline: Consistently achieves ~97% accuracy across text, image, and video.
- GPT-5 (Best Performer): Achieved 89.64% on text, but dropped to 66.96% on images and a lowly 56.72% on video.
- Open-Source Progress: The gap between open-source and proprietary models is closing. Models like InternVL-3.5-8B and Qwen3-VL-8B now perform comparably to lightweight commercial systems like GPT-4o-mini.
- Instruction Following: While large models are robust, smaller models (around 2B parameters) often "collapse" when faced with video inputs, with instruction-following accuracy dropping as low as 15%.
Limitations and Future Work
The authors acknowledge several constraints:
- Lexical Focus: The benchmark currently tests isolated, dictionary-level signs rather than continuous, conversational signing.
- Language Scope: It is limited to Chinese National Sign Language and does not yet cover regional dialects or other international sign languages (like ASL).
- Task Format: Multiple-choice testing is a diagnostic starting point; it doesn't evaluate a model's ability to generate sign language.
Future work will aim to expand the benchmark to multilingual settings, compositional sign understanding, and more open-ended, generative tasks.
Summary updated