CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language

Rui Zhao, Xuewen Zhong, Xiaoyun Zheng, Jinsong Su, Yidong Chen

TL;DR

Researchers have introduced CNSL-bench, the first comprehensive benchmark designed to evaluate how well Multimodal Large Language Models (MLLMs) actually understand Chinese National Sign Language. The results are a wake-up call: even top-tier models like GPT-5 struggle significantly with video-based sign language, trailing human performance by over 40 percentage points.

What Problem Does This Paper Address?

While Large Language Models (LLMs) have revolutionized natural language processing, sign language remains a "final frontier" for AI. Sign language isn't just a series of simple gestures; it is a complex, linguistically grounded system involving hand shapes, motion trajectories, and facial expressions.

Until now, most research has used LLMs as "black box" decoders at the end of a translation pipeline. We haven't really known if MLLMs possess the intrinsic ability to perceive and interpret the linguistic structure of signs directly from visual inputs. Furthermore, sign language varies by region, and the lack of a standardized, multimodal benchmark has made it impossible to compare different models fairly. CNSL-bench fills this gap by providing a massive, standardized dataset anchored to the official Chinese National Sign Language dictionary.

Methods

The authors constructed CNSL-bench using three core principles:

  1. Authoritative Grounding: They used the National Common Sign Language Dictionary to ensure every sign has a canonical, non-ambiguous definition.
  2. Multimodal Alignment: Every sign entry is represented in three ways: a textual description, an illustrative drawing, and a real-world video.
  3. Articulatory Diversity: The benchmark specifically categorizes signs into three linguistically distinct forms: air-writing (tracing characters in the air), finger-spelling (depicting character structures), and the manual-alphabet (Pinyin-based finger configurations).
Figure 1: Examples from CNSL-bench showing the aligned textual description, illustrative image, and corresponding sign language video for each sign entry.

To evaluate the models, the researchers used a four-way multiple-choice format. This "closed-form" design allows for scalable, objective testing, as current models are still too unreliable for open-ended sign language generation. They also employed Test-Time Scaling (Chain-of-Thought reasoning) to see if giving models more "time to think" would bridge the performance gap.

Datasets

  • CNSL-bench (New): The primary contribution, featuring 6,707 unique sign entries and 20,121 evaluation instances across text, image, and video modalities.
  • CNSL-DP: A large-scale video dataset used to provide the synchronized video recordings for the benchmark.
  • CSL-Daily: Used for a qualitative case study to show how flagship models fail at open-ended sentence-level sign understanding.

Key Insights and Ideas

  • The Modality Gap: Models are "text-heavy." They perform reasonably well when reading a description of a sign but fall apart when watching a video of the same sign. This suggests that visual-temporal grounding is the primary bottleneck.
  • Discrete vs. Continuous: Models are much better at finger-spelling (which is more discrete and character-like) than air-writing or manual-alphabets, which involve continuous, shape-intensive motions.
  • Reasoning Isn't a Silver Bullet: While "slow thinking" (Chain-of-Thought) helped models like Gemini-2.5-Flash, it actually led to negative gains for others like Qwen3-VL-Plus on certain tasks. Top-tier models also hit a "boundary effect" where more reasoning tokens eventually degraded performance in video settings.
  • Hallucination in Motion: Case studies revealed that when models fail at video, they don't just get it wrong—they hallucinate entirely different gestures (e.g., seeing a "typing" motion but interpreting it as a "crossing" gesture).
Figure 2: Examples illustrating the three categories of sign articulation: air-writing (top), finger-spelling (middle), and manual-alphabet (bottom).

Key Results

The study evaluated 21 state-of-the-art MLLMs. The quantitative findings highlight a massive disparity:

  • Human Baseline: Consistently achieves ~97% accuracy across text, image, and video.
  • GPT-5 (Best Performer): Achieved 89.64% on text, but dropped to 66.96% on images and a lowly 56.72% on video.
  • Open-Source Progress: The gap between open-source and proprietary models is closing. Models like InternVL-3.5-8B and Qwen3-VL-8B now perform comparably to lightweight commercial systems like GPT-4o-mini.
  • Instruction Following: While large models are robust, smaller models (around 2B parameters) often "collapse" when faced with video inputs, with instruction-following accuracy dropping as low as 15%.

Limitations and Future Work

The authors acknowledge several constraints:

  • Lexical Focus: The benchmark currently tests isolated, dictionary-level signs rather than continuous, conversational signing.
  • Language Scope: It is limited to Chinese National Sign Language and does not yet cover regional dialects or other international sign languages (like ASL).
  • Task Format: Multiple-choice testing is a diagnostic starting point; it doesn't evaluate a model's ability to generate sign language.

Future work will aim to expand the benchmark to multilingual settings, compositional sign understanding, and more open-ended, generative tasks.

Summary updated

Abstract

Sign language research has achieved significant progress due to the advances in large language models (LLMs). However, the intrinsic ability of LLMs to understand sign language, especially in multimodal contexts, remains underexplored. To address this limitation, we introduce CNSL-bench, the first comprehensive Chinese em{National Sign Language benchmark designed for evaluating multimodal large language models (MLLMs) in sign language understanding. The proposed CNSL-bench is characterized by: 1) Authoritative grounding, as it is anchored to the officially standardized \textit{National Common Sign Language Dictionary, mitigating ambiguity from regional or non-canonical variants and ensuring consistent semantic definitions; 2) Multimodal coverage, providing aligned textual descriptions, illustrative images, and sign language videos; and 3) Articulatory diversity, supporting fine-grained analysis across key manual articulatory forms, including air-writing, finger-spelling, and the Chinese manual-alphabet. Using CNSL-bench, we extensively evaluate 21 open-source and proprietary up-to-date MLLMs. Our results reveal that, despite recent advances in multimodal modeling, current MLLMs remain substantially inferior to human performance, exhibiting systematic disparities across input modalities and manual articulatory forms. Additional diagnostic analyses suggest that several performance limitations persist beyond improvements in reasoning and that instruction-following robustness varies substantially across models.
BibTeX
@inproceedings{zhao2026cnsl,
  author = {Rui Zhao and Xuewen Zhong and Xiaoyun Zheng and Jinsong Su and Yidong Chen},
  title = {{CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1896},
}

Loading related content...

Figure