Diffusion Models for Sign Language Video Anonymization

Zhaoyang Xia, Yang Zhou, Ligong Han, Carol Neidle, Dimitris N. Metaxas2024 · DOI

TL;DR

This paper introduces DiffSLVA, a novel method for anonymizing American Sign Language (ASL) videos using text-guided diffusion models. It addresses the critical need for privacy in ASL communication by altering signer identity while preserving linguistic content, enabling real-world applications for the Deaf community.

What Problem Does This Paper Address?

American Sign Language (ASL) is a rich, visual language that relies heavily on manual signs, facial expressions, and body movements to convey meaning. Unlike spoken languages, ASL lacks a standardized written form, making video the primary medium for communication and sharing among Deaf individuals. However, this reliance on video raises significant privacy concerns, as the identity of the signer is inherently visible. While anonymizing these videos is desirable for applications like anonymous peer review or data sharing, existing methods struggle to balance privacy with the preservation of crucial linguistic information, particularly subtle facial expressions. Furthermore, many current approaches require precise pose estimation and are limited to controlled datasets, hindering their application to real-world "in the wild" videos.

Methods

DiffSLVA leverages pre-trained large-scale diffusion models, specifically Stable Diffusion, to achieve zero-shot text-guided sign language video anonymization. The core innovation lies in adapting these image-generation models for video while incorporating mechanisms to maintain temporal consistency and preserve linguistic nuances.

Key methodological components include:

  • ControlNet Integration: To circumvent the need for explicit pose estimation, the authors integrate ControlNet. This allows the diffusion model to utilize low-level image features, such as Holistically-Nested Edge Detection (HED) edges, for guidance, making the process more robust to variations in "in the wild" videos.
  • Cross-Frame Attention: To ensure temporal consistency across video frames, the standard self-attention layers in the U-Net architecture of Stable Diffusion are replaced with cross-frame attention. This mechanism allows the model to reference previously generated frames and an anchor frame, promoting coherence.
  • Optical Flow Guided Latent Fusion: A two-stage optical flow guided latent fusion process is employed to further enhance frame consistency and prevent distortion during the denoising process.
  • Specialized Facial Expression Enhancement Module: Recognizing the critical role of facial expressions in ASL, a dedicated module is developed. This module utilizes an image animation model to specifically enhance the generation of faces, ensuring that the preserved linguistic content, particularly subtle expressions, is accurately conveyed. This module is integrated with a face parser for precise facial segmentation.
  • Text-Guided Anonymization: The entire process is guided by a text prompt, allowing users to specify the desired identity or style of the anonymized signer. This enables a high degree of control and customization.

Datasets

The authors utilized video datasets distributed through the American Sign Language Linguistic Research Project (ASLLRP). For evaluation, they selected signers of diverse genders and ages from these datasets, with each test sample limited to a maximum of 180 video frames. No new datasets were introduced.

Key Insights and Ideas

  • Zero-Shot Anonymization is Achievable: The paper demonstrates that it's possible to anonymize sign language videos without requiring specific training data for the anonymization task itself, relying instead on pre-trained diffusion models.
  • Low-Level Features are Key for "In the Wild" Robustness: By using ControlNet with HED edges, the method bypasses the brittle dependency on accurate pose estimation, which is often a bottleneck for real-world video processing.
  • Facial Expressions are Paramount: The explicit inclusion of a facial expression enhancement module highlights the critical importance of these non-manual features for preserving the linguistic integrity of ASL videos. Simply anonymizing the body is insufficient.
  • Text Prompts Offer Unprecedented Control: The ability to guide anonymization with text prompts opens up possibilities for diverse and personalized anonymization styles, catering to user preferences.

Key Results

The paper presents qualitative results demonstrating the effectiveness of DiffSLVA. The anonymized videos maintain clear hand shapes and movements, closely resembling the original signer's actions. The method successfully generates computer-generated signers with diverse appearances (e.g., a Superman, a CG style character, or an ink wash painting style) based on text prompts. An ablation study shows that the facial expression enhancement module significantly improves the preservation of linguistic meaning, particularly for nuanced expressions like topic and wh-question marking.

Limitations and Future Work

The authors acknowledge several limitations:

  • Face Occlusion and Blurring: The method may struggle in scenarios where the face is heavily occluded by hands or when there is significant blurring due to rapid sign movements.
  • Facial Expression Uniformity: While the enhancement module improves facial expression preservation, the underlying diffusion model can still tend to produce uniform expressions across frames, leading to some loss of linguistic meaning.
  • User Studies Needed: The full confirmation of the method's effectiveness for linguistic content transmission, complete identity disguise, and natural-looking signing requires user studies, which are planned for the future.

Future work will focus on addressing these limitations, particularly by further refining the facial transformation module and exploring more robust handling of challenging video conditions.

Summary updated

BibTeX
@misc{xia2024diffusion,
  author = {Zhaoyang Xia and Yang Zhou and Ligong Han and Carol Neidle and Dimitris N. Metaxas},
  title = {{Diffusion Models for Sign Language Video Anonymization}},
  year = {2024},
  doi = {10.63317/45yijutfouih},
}
Figure