vieCap4H Challenge 2021: A transformer-based method for Healthcare Image Captioning in Vietnamese

Doanh Bui Cao, Truc T. T. Trinh, Trong-Thuan Nguyen, Duc-Vu Nguyen, Nguyen Duy VoVNU 2022 · DOI

TL;DR

This paper presents a Transformer-based approach for generating Vietnamese captions for healthcare images, specifically for the VieCap4H challenge. By leveraging pre-trained language models like PhoBERT and a novel RSTNet architecture with an Adaptive Attention module, the team achieved competitive results, highlighting the importance of domain-specific language models and careful hyperparameter tuning.

What Problem Does This Paper Address?

Automatic image captioning, the task of generating descriptive text for images, is a fascinating intersection of Computer Vision and Natural Language Processing. While significant progress has been made, generating accurate and contextually relevant captions, especially for specialized domains like healthcare and in less common languages like Vietnamese, remains a challenge. Existing methods often rely on general-purpose models that may not capture the nuances of medical imagery or the specific linguistic structures of Vietnamese. The VieCap4H challenge aimed to address this gap by providing a Vietnamese healthcare image captioning dataset and task.

Methods

The proposed solution, VieCap4H, is built upon a Transformer-based architecture, specifically the RSTNet model. Here's a breakdown of their key methodological choices:

  • Image Representation: Instead of using complex region-based features, the authors opted for grid features. They adapted a method from previous work ([17]) that uses a ResNet backbone (specifically X101, X152, and X152++ variants) to extract features from the C5 block, resulting in a flattened grid of 49x2048 features per image. This approach balances computational efficiency with effectiveness.
  • Language Representation: Crucially, to handle the Vietnamese language, they pre-trained a BERT-based language model (BBLM) using the PhoBERT-base model. This is a significant departure from using generic word embeddings, as it allows the model to better understand the linguistic nuances of Vietnamese. They then employed a Masked Language Model (MLM) objective to combine visual signals with hidden states for language presentation.
  • RSTNet Architecture: The core of their system is the RSTNet model ([4]), which is a Transformer-based architecture. They specifically highlight two modules:
  • Grid Augmented (GA): This module incorporates information about the relative geometry between image grids into the Transformer's attention mechanism.
  • Adaptive Attention (AA): This module is designed to address cases where linguistic context is more important than visual features. It combines language presentations (from the BERT model), visual signals (from the encoder), and decoder hidden states to predict the next word. This allows the model to dynamically focus on relevant information.
  • Training Strategy: They employed a two-stage training approach:
  • Cross-Entropy Loss: Initial training for 20 epochs using standard cross-entropy loss.
  • Self-Critical Sequence Training (SCST): A single additional epoch of SCST training. SCST is a reinforcement learning-based technique that directly optimizes evaluation metrics like CIDEr. However, they found that excessive SCST training could degrade performance on this specific dataset, leading to the "20+1" schedule.

Datasets

  • VieCap4H Dataset: This is the primary dataset used for both training and evaluation, specifically designed for Vietnamese healthcare image captioning. The paper notes it contains 8032 samples.

Key Insights and Ideas

  • Domain-Specific Language Models Matter: Pre-training a BERT-based model on Vietnamese (PhoBERT) was crucial for achieving good performance, demonstrating that general language models are insufficient for specialized domains and languages.
  • Grid Features are Competitive: Despite the popularity of region-based features, the paper shows that well-tuned grid features can be effective and computationally more efficient for image captioning.
  • Adaptive Attention is Key: The Adaptive Attention module in RSTNet proved valuable by allowing the model to balance visual and linguistic information, leading to more contextually relevant captions.
  • SCST Tuning is Delicate: While SCST is a powerful technique for improving captioning metrics, its application requires careful tuning. The "20+1" schedule suggests that a little goes a long way for this specific task and dataset.
  • Hyperparameter Sensitivity: The VieCap4H dataset is highly sensitive to hyperparameters. Small changes in feature extraction, tokenization, and training/inference settings can significantly impact results, necessitating extensive experimentation.

Key Results

The team achieved strong performance in the VieCap4H challenge:

  • Public Test Round: An average BLEU score of 30.3%, ranking 3rd among participants.
  • Private Test Round: An average BLEU score of 28.9%, ranking 4th among participants.

These results were achieved using the X152++ backbone with specific hyperparameters (maxlength=23, beamsize=3). The paper also includes tables comparing different visual features, tokenizers, and training schedules, illustrating the impact of these choices on the BLEU score.

Limitations and Future Work

  • Feature Exploration: While grid features performed well, the authors experimented with region features and object features (from models trained on Visual Genome) but found them less effective. They speculate this is due to the domain mismatch.
  • Dataset Sensitivity: The paper emphasizes the extreme sensitivity of the VieCap4H dataset to various factors, suggesting that extensive hyperparameter tuning and feature selection are critical.
  • Future Directions: The authors plan to explore extracting features of specific objects, potentially combined with embedding vectors of object tags, to further improve results. They also aim to investigate other Transformer-based architectures and potentially more advanced pre-trained language models.

Summary updated

Abstract

The automatic image caption generation is attractive to both Computer Vision and Natural Language Processing research community because it lies in the gap between these two fields. Within the vieCap4H contest organized by VLSP 2021, we participate and present a Transformer-based solution for image captioning in the healthcare domain. In detail, we use grid features as visual presentation and pre-training a BERT-based language model from PhoBERT-base pre-trained model to obtain language presentation used in the Adaptive Decoder module in the RSTNet model. Besides, we indicate a suitable schedule with the self-critical training sequence (SCST) technique to achieve the best results. Through experiments, we achieve an average of 30.3% BLEU score on the public-test round and 28.9% on the private-test round, which ranks 3rd and 4th, respectively. Source code is available at https://github.com/caodoanh2001/uit-vlsp-viecap4h-solution.
BibTeX
@inproceedings{cao2022viecap,
  author = {Doanh Bui Cao and Truc T. T. Trinh and Trong-Thuan Nguyen and Duc-Vu Nguyen and Nguyen Duy Vo},
  title = {{vieCap4H Challenge 2021: A transformer-based method for Healthcare Image Captioning in Vietnamese}},
  booktitle = {VNU},
  year = {2022},
  doi = {10.25073/2588-1086/vnucsce.371},
}
Figure