Spotter+GPT: Turning Sign Spottings into Sentences with LLMs

Ozge Mercanoglu Sincan, Richard Bowden

TL;DR

This paper introduces Spotter+GPT, a novel framework for Sign Language Translation (SLT) that leverages Large Language Models (LLMs) like ChatGPT. By decoupling sign spotting from sentence generation, it significantly reduces the need for extensive SLT-specific training, offering a more accessible and efficient approach to translating sign language into spoken language.

What Problem Does This Paper Address?

Sign Language Translation (SLT) is a complex task aiming to convert sign language videos into spoken language sentences. Traditional SLT approaches often rely on end-to-end deep learning models that require vast amounts of meticulously annotated data, making them computationally expensive and time-consuming to train. Furthermore, sign languages have distinct grammatical structures compared to spoken languages, posing additional challenges for direct translation. Existing methods often struggle with large, weakly aligned datasets or require significant domain-specific training, limiting their practical application.

Methods

The Spotter+GPT framework adopts a two-stage, modular approach:

  1. Sign Spotting: The first stage uses a pre-trained Isolated Sign Language Recognition (ISLR) model, specifically an I3D model, to identify individual sign glosses within continuous sign language videos. This "spotter" is trained on a large dataset of isolated signs and then applied to video streams using a sliding window. The output is a sequence of detected glosses.
  2. Sentence Generation with LLM: The sequence of detected glosses is then fed into a Large Language Model (LLM), specifically ChatGPT, via prompting. The LLM is tasked with generating a coherent spoken language sentence based on these glosses. This stage leverages the LLM's natural language understanding and generation capabilities, bypassing the need for a dedicated gloss-to-text translation model.

A key innovation is the use of prompt engineering for ChatGPT. The authors refined the prompt to explicitly instruct the LLM to generate a meaningful sentence from the provided German words and to respond with "No Translation" if a sentence cannot be formed, ensuring more robust output.

Datasets

  • MeineDGS: A large, linguistic German Sign Language (DGS) dataset used for both training the sign spotter and evaluating the full SLT pipeline. The authors used a specific split (MeineDGS-V) and filtered glosses to create a vocabulary of 2,301 classes for the spotter.
  • DGS-20: A custom, smaller dataset of 20 German Sign Language videos collected by a single signer. This dataset was used for further evaluation, particularly to assess performance in a more controlled setting.

Key Insights and Ideas

  • LLMs as a powerful tool for SLT: The paper demonstrates that LLMs like ChatGPT can effectively bridge the gap between sign language glosses and spoken language sentences, even with imperfect gloss spotting.
  • Decoupling sign spotting and translation: By separating these two tasks, the framework avoids the need for end-to-end SLT training, significantly reducing computational costs and complexity.
  • Prompt engineering is crucial: The success of using LLMs for this task hinges on carefully crafted prompts that guide the model's behavior and handle potential errors or ambiguities.
  • Flexibility in handling imperfect inputs: ChatGPT shows a remarkable ability to generate semantically coherent sentences even when the input gloss sequence is incomplete or contains extraneous signs, highlighting its robustness.

Key Results

  • Sign Spotting Performance: The I3D spotter achieved competitive performance on the MeineDGS-V dataset, with per-instance accuracy of 54.57% and per-class accuracy of 42.48% after fine-tuning on both Kinetics and BOBSL datasets.
  • Spotter+GPT Performance:
    • On the MeineDGS-V test set, Spotter+GPT achieved a BLEURT score of 21.62. When using ground truth glosses (Sub-GT+GPT), the BLEURT score increased to 29.72, indicating the impact of spotter accuracy.
  • On the DGS-20 dataset, Spotter+GPT achieved a BLEURT score of 46.93, significantly outperforming a traditional Transformer baseline (19.85). This highlights the effectiveness of LLMs in more constrained, controlled environments.
  • Real-time Capability: The sign spotter can achieve real-time inference (minimum 25 frames per second) on systems with a GPU. ChatGPT's response time was observed to be around 1-2 seconds per prompt.

Limitations and Future Work

  • Spotter Accuracy Dependency: The overall translation quality is heavily reliant on the accuracy of the sign spotter. When the spotter fails to detect critical glosses, the translation quality degrades significantly.
  • Vocabulary Limitations: The vocabulary of the current spotter is limited to the 2,301 classes it was trained on.
  • Future Work: The authors suggest expanding the spotter's vocabulary to cover a wider range of recognized glosses. They also propose adapting the approach to specialized sign language interpretation by fine-tuning the spotter on domain-specific gloss sets. The system's ability to process live video inputs opens avenues for real-time applications.

Summary updated

Abstract

Sign Language Translation (SLT) is a challenging task that aims to generate spoken language sentences from sign language videos. In this paper, we introduce a lightweight, modular SLT framework, Spotter+GPT, that leverages the power of Large Language Models (LLMs) and avoids heavy end-to-end training. Spotter+GPT breaks down the SLT task into two distinct stages. First, a sign spotter identifies individual signs within the input video. The spotted signs are then passed to an LLM, which transforms them into meaningful spoken language sentences. Spotter+GPT eliminates the requirement for SLT-specific training. This significantly reduces computational costs and time requirements. The source code and pretrained weights of the Spotter are available at https://gitlab.surrey.ac.uk/cogvispublic/sign-spotter.
BibTeX
@inproceedings{sincan2025spotter,
  author = {Ozge Mercanoglu Sincan and Richard Bowden},
  title = {{Spotter+GPT: Turning Sign Spottings into Sentences with LLMs}},
  booktitle = {IVA (Adjunct)},
  year = {2025},
  doi = {10.1145/3742886.3756708},
}
Figure