Research on the Detection Method of Flight Trainees’ Attention State Based on Multi-Modal Dynamic Depth Network

Gongpu Wu, Changyuan Wang, Zehui Chen, Guangyi JiangMultimodal 2025 · DOI

TL;DR

This paper introduces M3D-Net, a novel lightweight, multi-modal deep learning architecture for detecting the attention state of flight trainees. By fusing temporal facial images, gaze data, and flight operation parameters, M3D-Net achieves a high accuracy of 97.56% with a compact model size, offering a promising solution for enhancing aviation safety.

What Problem Does This Paper Address?

Aviation safety is critically dependent on pilots maintaining a high level of attention to process complex, dynamic information. Human error, often stemming from attentional lapses, is a leading cause of aviation accidents. Traditional methods for assessing attention are often subjective or lack real-time capabilities, making them unsuitable for dynamic flight environments. Existing automated methods often rely on single data modalities or are computationally too heavy for practical deployment. This paper addresses the need for an accurate, efficient, and multi-modal approach to detect the attention states of flight trainees in a simulated environment.

Methods

The core of this work is the M3D-Net architecture, designed for lightweight and effective multi-modal attention detection. Key components include:

  • Multi-modal Feature Fusion: The system integrates three modalities:
  • Temporal Facial Images: Captured using an industrial camera.
  • Visual Gaze Information: Collected via a miniaturized eye-tracking device.
  • Flight Operation Data: Including parameters like altitude and pitch angle. An attention-based alignment mechanism is employed to enhance semantic correlations between these modalities, projecting them into a shared space and using multi-head self-attention for cross-modal relevance.

  • Lightweight Spatiotemporal Modeling: To efficiently capture temporal dynamics, the paper introduces a Depthwise Separable Convolution and LSTM (DSC-LSTM) module. This module replaces standard convolutions in ConvLSTM with depthwise separable convolutions, significantly reducing parameters while retaining the ability to model spatiotemporal dependencies.

  • Feature Extraction:
    • MobileNet-V3 is used for extracting features from temporal image sequences.
  • Bi-directional Long Short-Term Memory (Bi-LSTM) networks are used to process visual attention features (gaze data) and flight operation features, capturing their dynamic temporal context.

The M3D-Net architecture then fuses these features and uses the DSC-LSTM module for final spatiotemporal modeling before a fully connected layer for classification into six attention states.

Datasets

The study utilizes a newly constructed multimodal attention state dataset collected from flight cadets in a simulated flight environment. This dataset comprises:

  • Temporal Facial Images: 6348 samples, each containing 90 consecutive images (processed to 224x224 resolution).
  • Visual Attention Modality: 6348 sets of temporal eye movement feature vectors, including gaze coordinates, area, duration, and pupil diameter.
  • Flight Operation Modality: 6348 synchronous sets of flight control temporal data (altitude, pitch, yaw).

In total, the dataset contains 19,044 multimodal heterogeneous time-series samples, categorized into six attention states: Dangerous Behavior, Distracted Flying, Normal Level, High Concentration, Fatigue Flying, and Yawning. The data was split into training, testing, and validation sets (70:20:10).

Key Insights and Ideas

  • The Power of Multi-modal Fusion: The research strongly demonstrates that combining diverse data streams (visual, gaze, operational) significantly outperforms single-modal approaches for attention detection. This is crucial in complex domains like aviation where a single indicator might be insufficient.
  • Lightweight Architecture is Key: The M3D-Net's success hinges on its efficient design, particularly the DSC-LSTM module. This addresses the practical need for real-time deployment on potentially resource-constrained systems, a common challenge in embedded or operational environments.
  • Dynamic Temporal Modeling Matters: Moving beyond static analysis, the DSC-LSTM's ability to dynamically capture contextual dependencies within sequences (e.g., subtle shifts in gaze or operational patterns) is vital for accurately identifying nuanced attention states.
  • Attention for Alignment: The use of attention mechanisms for cross-modal feature alignment is a sophisticated way to ensure that information from different modalities is effectively integrated, enhancing semantic understanding.

Key Results

  • High Accuracy: The M3D-Net achieved an impressive 97.56% detection accuracy on the multimodal dataset.
  • Lightweight Design: The model boasts a small size of 18.6 M with 4.65 M parameters, significantly smaller than many comparable multimodal models.
  • Robustness: The confusion matrix and classification reports show high performance across most categories, with particularly strong recognition for "Dangerous Behavior" and "Yawning."
  • Ablation Studies: Comparisons with single-modal models (e.g., VGG19, EfficientNet) and other multimodal combinations (e.g., MobileNet + BiLSTM + ConvLSTM) highlight M3D-Net's superior performance and efficiency.
  • LOSO Cross-Validation: While performance slightly dropped in Leave-One-Subject-Out (LOSO) validation (indicating some inter-subject variability challenges), the model still maintained high accuracy (e.g., 0.9166 overall), demonstrating reasonable generalization.

Limitations and Future Work

  • Simulated Environment: The current dataset and experiments are based on a simulated flight environment. Real-world flight conditions can introduce additional complexities.
  • Environmental Interference: Factors like strong light stimulation and aircraft vibrations can potentially lead to false alarms by mimicking attention shifts.
  • Data Scarcity for Certain States: The "Fatigue" state had a relatively smaller sample size, which can sometimes impact model robustness for underrepresented classes.

Future work proposed by the authors includes:

  • Expanding data collection to real-world flight scenarios with diverse illumination and weather conditions.
  • Developing adaptive filtering algorithms to better distinguish environmental noise from genuine attention shifts.
  • Investigating adversarial sample generation techniques to further enhance model robustness against challenging conditions.
  • Exploring applications beyond aviation, such as automotive driving and medical monitoring.

Summary updated

Abstract

In aviation safety, pilots must efficiently process dynamic visual information and maintain a high level of attention. Any missed judgment of critical information or delay in decision-making may lead to mission failure or catastrophic consequences. Therefore, accurately detecting pilots’ attention states is the primary prerequisite for improving flight safety and performance. To better detect the attention state of pilots, this paper takes flight trainees as the research object and the simulated flight environment as the experimental background. It proposes a method for detecting the attention state of flight trainees based on a multi-modal dynamic depth network (M3D-Net). The M3D-Net architecture is a lightweight neural network architecture that integrates temporal image features, visual information features, and flight operation data features. It aligns image and text features through an attention mechanism to enhance the semantic association between modalities; it utilizes the Depth-wise Separable Convolution and LSTM (DSC-LSTM) module to model temporal information, dynamically capturing the contextual dependencies within the sequence, and achieving six-level attention state classification. This paper conducted ablation experiments to comparatively analyze the classification effects of the model and also evaluates the effectiveness of our proposed method through model evaluation metrics. Experiments show that the classification effect of the model architecture proposed in this paper reaches 97.56%, with a model size of 18.6 M. Compared with traditional algorithms, the M3D-Net architecture has better performance prospects in terms of application.
BibTeX
@inproceedings{wu2025research,
  author = {Gongpu Wu and Changyuan Wang and Zehui Chen and Guangyi Jiang},
  title = {{Research on the Detection Method of Flight Trainees’ Attention State Based on Multi-Modal Dynamic Depth Network}},
  booktitle = {Multimodal},
  year = {2025},
  doi = {10.3390/mti9100105},
}
Figure