Contrastive Token-Wise Meta-Learning for Unseen Performer Visual Temporal-Aligned Translation

Linjun Li, Tao Jin, Xize Cheng, Ye Wang, Wang Lin, Rongjie Huang, Zhou Zhao

Introduction

Visual temporal-aligned translation tasks, such as lipreading (visual speech recognition) and fingerspelling recognition, are critical for accessible communication and human-computer interaction. However, these systems often struggle when encountering "unseen performers"—individuals not present in the training data. This paper introduces the Contrastive Token-Wise Meta-learning (CtoML) framework, designed to enhance model generalization by addressing visual ambiguity and simulating domain shifts during training.

The Challenge of Performer Diversity

A primary obstacle in visual translation is the inherent ambiguity across different performers. Every individual possesses unique "performance habits": one speaker may exhibit subtle lip movements while another is highly expressive. In American Sign Language (ASL) fingerspelling, different signers may use slightly different handshapes for the same letter.

Visual ambiguity across letters, words, and performers.

Standard models tend to overfit to the specific habits of training performers, treating them as a monolithic domain. When tested on a new user, the domain shift leads to significant performance degradation. CtoML addresses this by treating each performer as a distinct domain and focusing on "ambiguous tokens"—words or letters that vary most across the population.

The CtoML Framework

The CtoML architecture integrates two innovative strategies into a Transformer-based backbone to achieve domain independence.

1. Token-Wise Diversity-Aware Weights

Recognizing that not all visual tokens are equally difficult to generalize, the authors developed a Token-wise Weights Calculator. This component measures the variance of visual representations for specific tokens across different performers. During the meta-training phase, the model is dynamically weighted to prioritize these high-variance, ambiguous tokens, forcing the decoder to learn more robust features.

2. Contrastive Meta-Learning

To simulate the experience of encountering new performers, the training process is divided into meta-train and meta-test stages.

  • Global Contrastive Loss: This ensures that the visual "prototype" for a specific word remains consistent in the semantic space, regardless of the performer's identity.
  • Local Contrastive Loss: This encourages the model to produce similar decoded outputs for the same token across different domains while maximizing the distance between representations of different tokens.
The overall CtoML framework architecture, highlighting the Visual Encoder, Cross-Modal Decoder, and Token-wise Weights Calculator.

Experimental Results

The CtoML framework was evaluated on two benchmark datasets: GRID (lipreading) and ChicagoFSWild (fingerspelling).

  • Lipreading (GRID): On the standard unseen speaker split, CtoML achieved a Word Error Rate (WER) of 5.42%, a significant improvement over the previous state-of-the-art (SOTA) model, AV-HuBERT, which recorded 6.64%.
  • Fingerspelling (ChicagoFSWild): In the challenging "in the wild" fingerspelling task, CtoML reached a Letter Accuracy of 54.9%, surpassing the previous best of 50.0%.

Ablation studies confirmed that both the diversity-aware weighting and the meta-learning strategy are critical. Removing either component resulted in a measurable drop in accuracy, demonstrating that the model's success stems from its ability to intelligently navigate visual ambiguity.

Conclusion

CtoML represents a significant step forward in making visual translation systems truly universal. By explicitly modeling the diversity of human performance and using meta-learning to bridge domain gaps, the framework achieves superior generalization to unseen performers. Future research will focus on removing the dependency on explicit performer identity labels, potentially through self-supervised clustering, to enable training on even larger, unlabeled datasets.

Summary updated

BibTeX
@inproceedings{li2023contrastive,
  author = {Linjun Li and Tao Jin and Xize Cheng and Ye Wang and Wang Lin and Rongjie Huang and Zhou Zhao},
  title = {{Contrastive Token-Wise Meta-Learning for Unseen Performer Visual Temporal-Aligned Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.699},
}
Figure