Automatic Detection of the Pharyngeal Phase in Raw Videos for the Videofluoroscopic Swallowing Study Using Efficient Data Collection and 3D Convolutional Networks

Jong Taek Lee, Eunhee Park, Tae-Du Jung

TL;DR

This paper introduces a novel, automated system for detecting the pharyngeal phase of swallowing in raw videofluoroscopic swallowing study (VFSS) videos. By employing a three-stage framework that efficiently collects training data and utilizes modified 3D convolutional networks, the system significantly reduces the manual effort and time required for VFSS analysis, achieving high detection accuracy with a low miss rate.

What Problem Does This Paper Address?

Videofluoroscopic Swallowing Study (VFSS) is a crucial diagnostic tool for dysphagia, a condition affecting millions, particularly the elderly and those with neurological disorders. A key challenge in VFSS analysis is identifying the pharyngeal phase, a rapid and critical stage where food passes from the mouth to the esophagus and the airway is protected. Currently, this identification relies on manual visual inspection of video frames, which is time-consuming, subjective, and prone to inter-observer variability. Existing automated methods often require manual annotations of anatomical landmarks or specific time intervals, adding to the complexity and labor. This paper aims to bridge this gap by developing an end-to-end system that automatically detects the pharyngeal phase in raw VFSS videos without requiring any manual annotations, thereby streamlining the diagnostic process.

Methods

The proposed framework is a three-stage pipeline:

  1. Pharyngeal Phase Candidate Generation: This stage leverages optical flow to identify short video clips exhibiting significant vertical motion, which is characteristic of bolus movement during the pharyngeal phase. An algorithm is used to generate candidate clips, aiming for high recall (92.0%) while accepting a moderate rate of false positives (50.1%) due to other actions like coughing.
  2. 3D Convolutional Network Training: The generated candidate clips are then fed into a 3D convolutional neural network (CNN) for classification. The authors use the Inflated Inception-V1 (I3D) architecture as a baseline and propose a modified version. This modified architecture incorporates more inception modules in earlier layers to better capture the rapid, subtle movements of the pharyngeal phase. Training is performed on both RGB and optical flow streams, and comparisons are made between models trained with random weights and those pre-trained on the Kinetics dataset.
  3. Pharyngeal Phase Detection: Finally, a sliding window technique is applied to raw, untrimmed VFSS videos. This technique uses the classification scores from the trained 3D CNN to detect the pharyngeal phase. A smoothing filter is applied to the classification scores, and a frame is marked as part of the pharyngeal phase if its score exceeds a threshold. The algorithm then identifies the start and end frames of detected phases, calculating a confidence score.

Datasets

The study utilized a VFSS dataset collected from 144 subjects at Kyungpook National University Chilgok Hospital. This dataset comprises raw VFSS videos recorded during swallowing studies, with subjects consuming various substances (thin liquid, thick liquid, semi-solid, solid) in different volumes. The videos were recorded at 30 frames per second (FPS) and sampled at 15 FPS for processing. The dataset was split into training (2696 clips) and validation (658 clips) sets for the video classification stage. For the detection stage, 215 raw VFSS videos were used. The authors also reference the Kinetics dataset for pre-training their models.

Key Insights and Ideas

  • Efficient Data Collection is Crucial: The paper highlights that training deep learning models for VFSS analysis is hampered by the difficulty and cost of manual annotation. Their optical flow-based candidate generation significantly alleviates this by automatically identifying potential pharyngeal phase segments, making data collection more efficient.
  • Modified CNNs for Subtle Motion: The pharyngeal phase involves rapid, complex movements that are difficult to capture with standard action recognition architectures. The proposed modification to the I3D network, by increasing the density of inception modules in early layers, demonstrates an effective way to enhance the network's sensitivity to these subtle temporal dynamics.
  • RGB vs. Optical Flow Trade-offs: While optical flow can capture motion information directly, it is more susceptible to noise and computationally more expensive than processing raw RGB frames. The study finds that RGB streams, despite longer training times, offer better accuracy and speed for this specific task, suggesting that the visual cues in RGB are sufficient and more robust.
  • Detection Performance is Key: The paper emphasizes that while classification accuracy is important, the ultimate goal is accurate detection of the pharyngeal phase in continuous video. The sliding window detection algorithm, combined with the classification scores, provides a practical solution for this.

Key Results

  • Improved Detection Performance: The proposed modified 3D CNN architecture, when trained with random weights and using RGB input, achieved the best performance among models trained from scratch, with a detection F1-score of 73.21%.
  • Reduced Analysis Time: The automated system significantly reduces the examination time for VFSS videos. The total video length for analysis was reduced from approximately 110 minutes to 10 minutes for the 215 test videos.
  • Low Miss Rate: The system achieved a low miss rate of 8.4% for detecting pharyngeal phase occurrences, meaning it correctly identified most instances.
  • Time Error: The detection time error was reported as 2.01 seconds for the best-performing model (proposed architecture with RGB and random weights), which is an improvement over baseline methods.
  • Pre-trained Models Advantage: Pre-trained models (specifically Inception-V1/RGB/pre-trained) showed the highest detection F1-score (84.25%) but required more computational resources for training. The proposed model is presented as a strong alternative when resources are limited.

Limitations and Future Work

  • Limited Scope: The current framework focuses solely on detecting the pharyngeal phase. The authors acknowledge that detecting the oral and esophageal phases, as well as pre-swallow or post-swallow aspiration events, are important extensions for future work.
  • Detection Time Error: While the system reduces overall analysis time, the detection time error (around 2 seconds) is still a factor. Future research will focus on further reducing this error.
  • Aspiration Detection: The paper does not directly detect aspiration events, which are a critical clinical outcome. Future work could integrate the pharyngeal phase detection with aspiration detection mechanisms.
  • Manual Annotations for Ground Truth: While the system avoids manual annotations for training, the ground truth for evaluation still relies on expert clinician annotations, which can have inherent variability.

Summary updated

Abstract

Videofluoroscopic swallowing study (VFSS) is a standard diagnostic tool for dysphagia. To detect the presence of aspiration during a swallow, a manual search is commonly used to mark the time intervals of the pharyngeal phase on the corresponding VFSS image. In this study, we present a novel approach that uses 3D convolutional networks to detect the pharyngeal phase in raw VFSS videos without manual annotations. For efficient collection of training data, we propose a cascade framework which no longer requires time intervals of the swallowing process nor the manual marking of anatomical positions for detection. For video classification, we applied the inflated 3D convolutional network (I3D), one of the state-of-the-art network for action classification, as a baseline architecture. We also present a modified 3D convolutional network architecture that is derived from the baseline I3D architecture. The classification and detection performance of these two architectures were evaluated for comparison. The experimental results show that the proposed model outperformed the baseline I3D model in the condition where both models are trained with random weights. We conclude that the proposed method greatly reduces the examination time of the VFSS images with a low miss rate.
BibTeX
@inproceedings{lee2019automatic,
  author = {Jong Taek Lee and Eunhee Park and Tae-Du Jung},
  title = {{Automatic Detection of the Pharyngeal Phase in Raw Videos for the Videofluoroscopic Swallowing Study Using Efficient Data Collection and 3D Convolutional Networks}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=fNo7K1aoM9},
}
Figure