SignIT: A Comprehensive Dataset and Multimodal Analysis for Italian Sign Language Recognition

Alessia Micieli, Giovanni Maria Farinella, Francesco Ragusa

TL;DR

Researchers have introduced SignIT, a new multimodal dataset specifically designed to advance Italian Sign Language (LIS) recognition. By providing 644 annotated videos across 94 sign classes and extracting detailed 2D pose keypoints, the authors demonstrate that while current state-of-the-art models can categorize broad topics (like "Animals" vs. "Food"), they still struggle significantly with the fine-grained nuances required to distinguish between specific, visually similar signs.

What Problem Does This Paper Address?

While sign languages are the primary mode of communication for millions, automated recognition technology is far from universal. Most research has focused on American Sign Language (ASL) or Chinese Sign Language (CSL), leaving Italian Sign Language (LIS) significantly under-represented.

The primary bottleneck for LIS is a lack of high-quality, annotated data. Existing LIS datasets are often small-scale or limited to "fingerspelling" (spelling out words letter-by-letter). To bridge this gap, the authors developed SignIT to support "isolated sign recognition"—the ability to identify a complete concept or word from a video clip—which is a critical step toward full, continuous sign-to-speech translation.

Methods

The authors took a multimodal approach, combining traditional computer vision with modern Large Language Models (LLMs). Their workflow included:

  1. Data Acquisition & Preprocessing: They collected 644 videos from 45 different signers on YouTube. To ensure models weren't "cheating" by reading on-screen text or background cues, they applied a blurring operation to those specific regions.
  2. Keypoint Extraction: Using Google’s MediaPipe, they extracted 2D coordinates for 21 hand points, 51 facial landmarks, and 33 body joints per frame. This transforms raw video into a "skeleton" that captures the geometry of the sign.
  3. Benchmarking Baselines: They tested several architectures:
  4. K-NN & MLP: Simple models trained purely on the extracted 2D keypoints.
  5. ResNet18 & I3D: Standard vision models that process raw RGB frames or short video clips.
  6. LLaVA-OneVision: A state-of-the-art multimodal LLM. They tested this in zero-shot, fine-tuned, and "pose-aware" configurations (where keypoints are projected onto the image).

Datasets

The core contribution is the SignIT Dataset:

  • Scale: 644 videos, 3.33 hours of footage, and ~99,000 annotated frames.
  • Taxonomy: 94 distinct signs organized into 5 macro-categories: Animals, Food, Colors, Emotions, and Family.
  • Diversity: Recorded by 45 different users in 37 different indoor environments, providing a robust test for background and signer invariance.

The paper also provides a comprehensive comparison against existing benchmarks like PHOENIX14T (German), WLASL (American), and the original LIS Corpus.

Key Insights and Ideas

  • Macro vs. Micro Difficulty: Models are surprisingly good at identifying the category of a sign (e.g., knowing a sign belongs to "Animals"). However, performance plummets when asked to identify the specific sign (e.g., distinguishing "Dog" from "Horse").
  • The Power of Keypoints: For broad categorization, simple Multi-Layer Perceptrons (MLPs) using only 2D keypoints actually outperformed complex vision models. This suggests that the "geometry" of the gesture is often more discriminative than the visual texture of the video.
  • Visual Ambiguity: The authors highlight "minimal pairs" in sign language—signs that are identical except for a tiny movement or facial expression (like "Green" vs. "Purple"). Current models lack the spatial resolution or temporal sensitivity to catch these subtle differences reliably.
  • Fine-tuning is Non-Negotiable: For the LLaVA-OneVision model, fine-tuning on the SignIT dataset improved accuracy from a dismal 13.8% to 63.5% for macro-categories, proving that general-purpose multimodal models need domain-specific "instruction" for sign language.

Key Results

  • Macro-Category Performance: The MLP model achieved the best overall F1-score of 0.706, followed closely by ResNet-18 (0.646) and fine-tuned LLaVA-OneVision (0.644).
  • Micro-Category (94 signs) Performance: This task proved much harder. The best-performing model (fine-tuned LLaVA-OneVision with Pose) only reached an accuracy of 0.152 and an F1-score of 0.206.
  • Temporal Importance: For fine-grained recognition, using a sequence of frames (Sequence) significantly outperformed using a single central frame, highlighting that the "motion path" is essential for distinguishing complex signs.

Limitations and Future Work

The authors acknowledge several hurdles:

  • The Long Tail: The dataset has a "long-tail" distribution, meaning some signs (like "Spider" or "Snake") have very few examples, making it difficult for models to learn them.
  • Visual Similarity: Models still struggle with signs that share similar handshapes or locations.
  • Continuous Signing: This dataset focuses on isolated signs. Real-world LIS involves continuous signing with complex grammar and "co-articulation" (where one sign blends into the next), which remains a massive open challenge.

The team has released the data and annotations publicly to encourage the community to tackle these limitations.

Summary updated

Abstract

In this work we present SignIT, a new dataset to study the task of Italian Sign Language (LIS) recognition. The dataset is composed of 644 videos covering 3.33 hours. We manually annotated videos considering a taxonomy of 94 distinct sign classes belonging to 5 macro-categories: Animals, Food, Colors, Emotions and Family. We also extracted 2D keypoints related to the hands, face and body of the users. With the dataset, we propose a benchmark for the sign recognition task, adopting several state-of-the-art models showing how temporal information, 2D keypoints and RGB frames can be influence the performance of these models. Results show the limitations of these models on this challenging LIS dataset. We release data and annotations at the following link: https://fpv-iplab.github.io/SignIT/.
BibTeX
@inproceedings{micieli2026signit,
  author = {Alessia Micieli and Giovanni Maria Farinella and Francesco Ragusa},
  title = {{SignIT: A Comprehensive Dataset and Multimodal Analysis for Italian Sign Language Recognition}},
  booktitle = {VISAPP},
  year = {2026},
  doi = {10.5220/0014480100004084},
}
Figure