First Trimester Video Saliency Prediction Using Clstmu-Net with Stochastic Augmentation

Elizaveta Savochkina, Lok Hin Lee, He Zhao, Lior Drukker, Aris T. Papageorghiou, J. Alison Noble

Abstract

In this paper we develop a multi-modal video analysis algorithm to predict where a sonographer should look next. Our approach uses video and expert knowledge, defined by gaze tracking data, which is acquired during routine first-trimester fetal ultrasound scanning. Specifically, we propose a spatio-temporal convolutional LSTMU-Net neural network (cLSTMU-Net) for video saliency prediction with stochastic augmentation. The architecture design consists of a U-Net based encoder-decoder network and a cLSTM to take into account temporal information. We compare the performance of the cLSTMU-Net alongside spatial-only architectures for the task of predicting gaze in first trimester ultrasound videos. Our study dataset consists of 115 clinically acquired first trimester US videos and a total of 45, 666 video frames. We adopt a Random Augmentation strategy (RA) from a stochastic augmentation policy search to improve model performance and reduce over-fitting. The proposed cLSTMU-Net using a video clip of 6 frames outperforms the baseline approach on all saliency metrics: KLD, SIM, NSS and CC (2.08, 0.28, 4.53 and 0.42 versus 2.16, 0.27, 4.34 and 0.39).
BibTeX
@inproceedings{savochkina2022first,
  author = {Elizaveta Savochkina and Lok Hin Lee and He Zhao and Lior Drukker and Aris T. Papageorghiou and J. Alison Noble},
  title = {{First Trimester Video Saliency Prediction Using Clstmu-Net with Stochastic Augmentation}},
  booktitle = {ISBI},
  year = {2022},
  doi = {10.1109/isbi52829.2022.9761585},
}

Loading related content...

Figure