Sensor Data Fusion Based on Deep Learning for Computer Vision Applications and Medical Applications

Rizwan Ali Naqvi, Muhammad Arsalan, Talha Qaiser, Tariq Mahmood Khan, Imran Razzak

TL;DR

This editorial introduces a special issue of Sensors focused on sensor data fusion for computer vision and medical applications, highlighting the growing importance of deep learning in this domain. It showcases recent advancements in fusing data from various sensors to improve accuracy and efficiency in tasks ranging from activity recognition to disease diagnosis.

What Problem Does This Paper Address?

The paper addresses the increasing complexity and demand for high-accuracy perception systems in computer vision and medical applications. Traditional methods often struggle with environmental variations and the inherent limitations of single sensors. Sensor data fusion, the process of combining information from multiple sensors, offers a way to overcome these limitations by providing more robust and reliable data. However, effectively integrating and processing this fused data, especially using advanced techniques like deep learning, presents significant challenges. This special issue aims to explore cutting-edge research that tackles these challenges, particularly in areas like spoof detection, image analysis, and medical diagnostics.

Methods

The editorial highlights several deep learning-based approaches for sensor data fusion. These include:

  • 3DMesh-Group Activity Recognition (GAR): A method that uses 3D body meshes derived from various sensor inputs (including visual data) to recognize multi-person activities, aiming for improved functional, cognitive, and physical health monitoring.
  • Convolutional Neural Networks (CNNs) for Semantic Segmentation: Specifically, the "DeeperLabC" model is discussed for its application in underwater monitoring to detect and segment corals, aiding in pollution assessment. This model utilizes an encoder-decoder architecture and ResNet34 for feature extraction.
  • YOLO-based Human Count Tracker: Employed in conjunction with visual sentiment analysis of disaster images to monitor individuals and estimate risk.
  • Hybrid Deep Feature Extraction and Classification: Techniques like SARVOTAM (Summing of neurAl aRchitecture and Visualization Technology for Android Malware identification) convert malware features into images for CNN-based classification.
  • Recurrent Neural Networks (RNNs): Used for forecasting biomarkers in Alzheimer's disease progression.
  • Deep Learning for Medical Image Analysis: Various CNN architectures (eNet, AlexNet, VGG16, ResNet-50, Inception-V1) are compared for lung cancer detection.
  • Hybrid Deep Features and Extreme Learning Machines: Applied to multiclass skin lesion classification.
  • Deep Learning for Diabetic Retinopathy Detection: A CNN model with a hybrid feature-embedding technique is proposed for early detection of microaneurysms.
  • Bluetooth Low Energy (BLE) for Contact Tracing: A sigmoid-based model is used to categorize contact data risk for COVID-19 mitigation.

Datasets

The following datasets are mentioned:

  • CoralS: Used for training the DeeperLabC model for coral segmentation.
  • Social Media Platforms: Datasets of images related to natural catastrophes were collected for sentiment and activity analysis.
  • DREBIN: Used for the experiment on android malware image classification.
  • "Poha" Dataset: A 44-character Pashto handwritten character dataset.
  • LUNA16: A publicly available dataset used for lung cancer detection.
  • Alzheimer's Disease Neuroimaging Initiative (ADNI): An openly available dataset used for Alzheimer's disease biomarker prediction.
  • HAM10000 and ISIC2018: Publicly available datasets used for multiclass skin lesion classification.
  • E-Ophtha and DIARETDB1: Publicly accessible datasets used for microaneurysm detection in diabetic retinopathy.

Key Insights and Ideas

  • Deep learning is a powerful enabler for sensor data fusion: The editorial consistently emphasizes how deep learning techniques are crucial for processing and merging complex data from diverse sensors, leading to improved performance.
  • Sensor fusion is vital for robust perception: By combining data from multiple sources, systems can achieve higher accuracy and reliability, overcoming the limitations of individual sensors in challenging environments.
  • Applications span critical domains: The presented research highlights the broad applicability of sensor data fusion, from everyday computer vision tasks like activity recognition to life-saving medical diagnostics.
  • Addressing real-world challenges: The special issue showcases solutions for complex problems such as identifying subtle medical conditions (diabetic retinopathy, skin cancer), detecting sophisticated threats (android malware), and monitoring public health (COVID-19 contact tracing).

Key Results

  • Coral Segmentation: DeeperLabC achieved an F1 score of 97.10% and a mean IoU of 93.90% for coral segmentation.
  • Android Malware Classification: The proposed CNN-SVM model achieved an accuracy of 92.59% using android certificates and manifest malware images.
  • Pashto Handwritten Character Recognition: The proposed CNN model achieved 99.64% accuracy for Pashto handwritten character recognition.
  • Lung Cancer Detection: AlexNet with the SGD optimizer achieved a precision of 97.42%.
  • Alzheimer's Disease Prediction: The proposed recurrent neural network model achieved an accuracy of 88.24%.
  • Skin Lesion Classification: Accuracies of 93.40% and 94.36% were attained on the HAM10000 and ISIC2018 datasets, respectively.
  • Diabetic Retinopathy Detection: The proposed method achieved classification accuracies of 96% for E-Ophtha and 94% for DIARETDB1.

Limitations and Future Work

  • Limited studies on fusion architecture: The editorial notes that there are few studies specifically on sensor and data fusion architectures for cooperative perception and risk assessment in computer vision and medical applications.
  • Need for high-quality perception: Real-time, high-quality perception mechanisms are necessary for deploying computer vision and deep learning applications effectively.
  • Challenges in computer vision: The performance of computer vision technology is still challenged by external environmental factors, requiring further improvement.
  • Future directions implied by the special issue: The special issue itself aims to explore high-caliber research in areas like multiple-approach fusion, spoof detection, image detection, localization, classification, and segmentation, suggesting these are active and important areas for future work.

Summary updated

BibTeX
@inproceedings{naqvi2022sensor,
  author = {Rizwan Ali Naqvi and Muhammad Arsalan and Talha Qaiser and Tariq Mahmood Khan and Imran Razzak},
  title = {{Sensor Data Fusion Based on Deep Learning for Computer Vision Applications and Medical Applications}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=yiWC6zUINF},
}

Loading related content...

Figure