Character-level Japanese Text Generation with Attention Mechanism for Chest Radiography Diagnosis

Kenya Sakka, Kotaro Nakayama, Nisei Kimura, Taiki Inoue, Yusuke Iwasawa, Ryohei Yamaguchi, Yoshimasa Kawazoe, Kazuhiko Ohe, Yutaka MatsuoarXiv 2020 · arXiv

Introduction

Automated chest radiography diagnosis has advanced significantly, primarily within English-speaking contexts. However, applying these technologies to the Japanese language introduces unique linguistic challenges. Unlike English, Japanese lacks explicit word boundaries (spaces), making standard tokenization prone to errors, particularly with specialized medical terminology. Furthermore, Japanese frequently employs multiple orthographic variants for the same concept (e.g., mixing Kanji and Katakana). Traditional word-level models treat these variants as distinct entities, which dilutes training data and reduces model robustness.

To address these issues, this research proposes an end-to-end deep learning model that generates Japanese medical findings directly from chest X-rays using a character-level approach. This method bypasses the need for complex segmentation and handles orthographic variations more effectively.

Methodology

The proposed system utilizes an Encoder-Decoder architecture integrated with a "soft-attention" mechanism.

Encoder and Decoder Design

The encoder uses a ResNet-151 CNN to extract visual features from the input radiographs. To ensure the encoder recognizes domain-specific medical features, it was pre-trained on a classification task using over 13,000 chest images before being integrated into the generation pipeline.

The decoder is a Long Short-Term Memory (LSTM) network. Unlike conventional models that predict the next word, this decoder predicts the next character. This character-level generation allows the model to construct technical terms and handle spelling variations without a predefined vocabulary dictionary.

Architecture of the proposed Encoder-Decoder model with Attention mechanism.

Attention Mechanism

The attention layer enables the decoder to dynamically focus on specific regions of the X-ray image while generating each character. This visual grounding provides interpretability, allowing radiologists to see which parts of the image influenced the generated text.

Dataset and Experimental Setup

The study utilized the JSRT (Japanese Society of Radiological Technology) Dataset, comprising 18,004 pairs of frontal chest radiographs and Japanese findings. Two subsets were created based on frequency thresholds (findings appearing at least 5 times vs. 30 times). To mitigate the class imbalance where approximately 75% of images are "normal," the researchers employed oversampling and undersampling techniques.

Results and Discussion

The model's performance was evaluated using BLEU scores and qualitative visual inspection of attention maps.

Quantitative Performance

The model achieved a high BLEU-4 score of 0.8214 on the original distribution, largely due to its high accuracy in identifying "normal" cases. However, when tested exclusively on abnormal images, the BLEU-4 score dropped to approximately 0.1713. This discrepancy highlights the complexity of describing diverse pathologies compared to the relative consistency of normal reports.

Qualitative Analysis and Visual Grounding

Qualitative results demonstrate that the attention mechanism successfully "tracks" pathologies. For instance, when generating terms for a pacemaker or diaphragm elevation, the attention weights correctly localized to the relevant anatomical or device regions.

Qualitative results showing generated findings and corresponding attention maps for various pathologies.

Key Insights

  1. Characters over Words: Operating at the character level eliminates the need for a dictionary and improves the handling of technical terms like "両肺" (both lungs) versus "両側肺".
  2. Domain-Specific Pre-training: Pre-training the encoder on medical images is essential; general ImageNet pre-training is insufficient for the nuances of chest radiography.

Conclusion and Future Work

While the model successfully generates interpretable Japanese findings, it faces limitations. The inherent "normal" bias in medical datasets remains a challenge, and the model currently lacks the ability to perform temporal comparisons with previous scans—a common clinical requirement. Future research will focus on developing evaluation metrics that prioritize clinical correctness over linguistic similarity, as even minor linguistic errors can have significant clinical implications.

Summary updated

Abstract

Chest radiography is a general method for diagnosing a patient's condition and identifying important information; therefore, radiography is used extensively in routine medical practice in various situations, such as emergency medical care and medical checkup. However, a high level of expertise is required to interpret chest radiographs. Thus, medical specialists spend considerable time in diagnosing such huge numbers of radiographs. In order to solve these problems, methods for generating findings have been proposed. However, the study of generating chest radiograph findings has primarily focused on the English language, and to the best of our knowledge, no studies have studied Japanese data on this subject. There are two challenges involved in generating findings in the Japanese language. The first challenge is that word splitting is difficult because the boundaries of Japanese word are not clear. The second challenge is that there are numerous orthographic variants. For deal with these two challenges, we proposed an end-to-end model that generates Japanese findings at the character-level from chest radiographs. In addition, we introduced the attention mechanism to improve not only the accuracy, but also the interpretation ability of the results. We evaluated the proposed method using a public dataset with Japanese findings. The effectiveness of the proposed method was confirmed using the Bilingual Evaluation Understudy score. And, we were confirmed from the generated findings that the proposed method was able to consider the orthographic variants. Furthermore, we confirmed via visual inspection that the attention mechanism captures the features and positional information of radiographs.
BibTeX
@misc{sakka2020character,
  author = {Kenya Sakka and Kotaro Nakayama and Nisei Kimura and Taiki Inoue and Yusuke Iwasawa and Ryohei Yamaguchi and Yoshimasa Kawazoe and Kazuhiko Ohe and Yutaka Matsuo},
  title = {{Character-level Japanese Text Generation with Attention Mechanism for Chest Radiography Diagnosis}},
  year = {2020},
  eprint = {2004.13846},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.13846},
}
Figure