The TUB Sign Language Corpus Collection

Eleftherios Avramidis, Vera Czehmann, Fabian Deckert, Lorenz Hufe, Aljoscha Lipski, Yuni Amaloa Quintero Villalobos, Tae Kwon Rhee, Mengqian Shi, Lennart Stölting, Fabrizio Nunnari, Sebastian MöllerIva (adjunct) 2025 · DOI

TL;DR

This paper introduces the TUB Sign Language Corpus Collection, a massive, openly available dataset of 12 sign languages with synchronized spoken language subtitles. This resource aims to bridge the significant gap in sign language data, enabling advancements in sign language technology and improving accessibility for the deaf and hard-of-hearing community.

What Problem Does This Paper Address?

Despite rapid progress in spoken language technology, sign languages have been largely left behind. This disparity creates a significant disadvantage for the estimated half-billion sign language users worldwide, limiting their access to information, communication technologies, and the benefits of AI-driven tools. A primary bottleneck is the scarcity of large, high-quality, and accessible sign language corpora, especially parallel corpora that link sign language with its corresponding spoken language. Existing datasets are often small, cover only a few languages, or lack crucial metadata and licensing clarity, hindering research and development.

Methods

The creation of the TUB Sign Language Corpus Collection involved a multi-stage process:

  • Manual Data Collection: Eight native speakers of the respective spoken languages scoured online sources, primarily YouTube, news broadcasts, government channels, and educational platforms, to identify relevant sign language content. This manual approach ensured a more targeted and contextually aware selection compared to purely automated methods.
  • Licensing Verification and Permissions: A critical step was meticulously verifying the licensing of all collected content. The authors also proactively contacted content creators to inform them about the project and obtain explicit permissions, ensuring ethical data usage.
  • Content Processing: Subtitles were extracted from videos, often using Optical Character Recognition (OCR) for embedded captions. Sentence and word tokenization were applied to these subtitles. Videos were also cropped to isolate the sign language interpreter, improving the quality and focus of the visual data.
  • Metadata Generation: Comprehensive metadata was compiled, including channel information, video details, licensing, and statistics on subtitles, sentences, tokens, and characters. This metadata is provided in two levels: a channel list and a detailed video list.

Datasets

The paper introduces the TUB Sign Language Corpus Collection, which comprises:

  • 1,391 hours of continuous sign language videos.
  • 4,381 video files in total.
  • 1.3 million timed closed captions (subtitles) across the videos, containing approximately 14 million tokens.
  • The collection covers 12 sign languages, with a significant focus on providing the first consistent parallel corpora for 8 Latin American sign languages.

This collection is made available as an open-source catalogue under the MIT license. The authors highlight that their German Sign Language (DGS) corpora are ten times larger than previously available DGS corpora.

Key Insights and Ideas

  • The Criticality of Parallel Corpora: The paper strongly emphasizes that the lack of parallel corpora is a major impediment to sign language technology research, mirroring the foundational role such corpora played in spoken language NLP.
  • Ethical Data Sourcing is Paramount: The authors' commitment to verifying licenses and obtaining explicit permissions from content creators sets a strong precedent for responsible data collection in this domain.
  • Bridging the Latin American Sign Language Gap: The introduction of consistent parallel corpora for eight Latin American sign languages is a significant contribution, addressing a severe underrepresentation in existing resources.
  • The Value of Manual Curation: While time-consuming, the manual selection process by native speakers allowed for better identification of relevant content and understanding of cultural context, leading to a higher quality dataset.

Key Results

The TUB Sign Language Corpus Collection is a substantial resource, totaling 1,391 hours of video and 14 million tokens in subtitles. The collection includes data for 12 sign languages, with notable contributions in German Sign Language (DGS) and a pioneering set for eight Latin American sign languages. The metadata provided allows for detailed analysis and use in various downstream tasks.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dominance of Hearing Curators: The majority of the data collection and annotation was performed by hearing individuals, with only one DGS signer involved. This could impact the identification of nuances and quality of captions.
  • Lack of Local Community Involvement: Direct engagement with local signing communities was not possible during the development phase, which is crucial for ensuring FAIR and CARE principles.
  • Subtitle vs. Sign Language: The provided subtitles are transcriptions of the spoken language, not direct translations of the sign language itself, meaning there can be deviations.
  • No Visual Interpreter Isolation: Currently, isolated video of the signing interpreter is not provided due to licensing restrictions.

Future work will focus on:

  • Increasing the data volume and availability.
  • Improving temporal alignment of subtitles with signed content.
  • Enhancing community engagement to better serve signing communities.
  • Potentially providing co-ordinates and scripts for visual data in future updates.

Summary updated

BibTeX
@inproceedings{avramidis2025tub,
  author = {Eleftherios Avramidis and Vera Czehmann and Fabian Deckert and Lorenz Hufe and Aljoscha Lipski and Yuni Amaloa Quintero Villalobos and Tae Kwon Rhee and Mengqian Shi and Lennart Stölting and Fabrizio Nunnari and Sebastian Möller},
  title = {{The TUB Sign Language Corpus Collection}},
  booktitle = {Iva (adjunct)},
  year = {2025},
  doi = {10.1145/3742886.3756709},
}
Figure