Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition

Tong Yu, Didier Mutter, Jacques Marescaux, Nicolas PadoyarXiv 2018 · OpenReview

Abstract

Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior. In laparoscopic procedures one particular algorithm needed for such systems is the identification of surgical phases, for which the current state of the art is a model based on a CNN-LSTM. A number of previous works using models of this kind have trained them in a fully supervised manner, requiring a fully annotated dataset. Instead, our work confronts the problem of learning surgical phase recognition in scenarios presenting scarce amounts of annotated data (under 25% of all available video recordings). We propose a teacher/student type of approach, where a strong predictor called the teacher, trained beforehand on a small dataset of ground truth-annotated videos, generates synthetic annotations for a larger dataset, which another model - the student - learns from. In our case, the teacher features a novel CNN-biLSTM-CRF architecture, designed for offline inference only. The student, on the other hand, is a CNN-LSTM capable of making real-time predictions. Results for various amounts of manually annotated videos demonstrate the superiority of the new CNN-biLSTM-CRF predictor as well as improved performance from the CNN-LSTM trained using synthetic labels generated for unannotated videos. For both offline and online surgical phase recognition with very few annotated recordings available, this new teacher/student strategy provides a valuable performance improvement by efficiently leveraging the unannotated data.

Introduction: The Annotation Bottleneck

For computer-assisted surgery (CAS) systems to provide context-aware assistance—such as displaying relevant anatomical information or predicting the next surgical step—they must understand the current phase of an operation in real-time. While deep learning architectures like CNN-LSTMs have shown great promise in surgical phase recognition, their performance is heavily dependent on large, manually annotated datasets.

Labeling surgical video is an arduous task requiring significant clinical expertise, creating a major bottleneck in the development of these systems. Researchers often possess vast amounts of raw video data but only a small fraction of manual annotations. This paper addresses this challenge by asking: how can we effectively leverage unannotated videos to improve real-time recognition models when manual labels are scarce?

Methodology: The Teacher/Student Framework

The authors introduce a two-stage semi-supervised framework designed to maximize the utility of limited ground-truth data. The core of this approach is the interaction between two distinct models:

  1. The Teacher (CNN-biLSTM-CRF): This is a high-capacity "offline" model. It utilizes a ResNet-50 CNN for visual feature extraction, a bidirectional LSTM (biLSTM) to incorporate both past and future temporal context, and a Conditional Random Field (CRF) to ensure chronological consistency in phase transitions. While its reliance on "future" frames prevents real-time use, it serves as a highly accurate annotator.
  2. The Student (CNN-LSTM): This is a standard "online" model suitable for real-time deployment. It processes only current and past frames, making it inherently less accurate than the teacher but capable of live surgical assistance.
The proposed teacher/student workflow: (1) Train teacher on manual labels, (2) Teacher annotates unlabeled videos, (3) Merge datasets, (4) Train student on the combined pool.

As illustrated in the workflow above, the Teacher is first trained on a small set of manually annotated videos. It then generates "synthetic labels" for the remaining unannotated pool. Finally, the Student is trained on the combined dataset of manual and synthetic annotations, allowing it to learn from the structural insights captured by the more powerful Teacher model.

Experimental Setup

The framework was evaluated using the Cholec120 dataset, which consists of 120 laparoscopic cholecystectomy videos labeled with seven surgical phases. To simulate data-scarce environments, the authors tested the model using "mini-sets" of only 1, 3, 5, 10, and 20 manually labeled videos from a training pool of 80.

Results: Bridging the Data Gap

The experimental results highlight two primary findings:

  • Teacher Superiority: The CNN-biLSTM-CRF architecture consistently outperformed standard models. With only 20 labeled videos, it achieved an F1 score of 75.8%, compared to 70.2% for a standard CNN-LSTM trained on the same data.
  • Student Improvement via Synthetic Labels: When the Student (CNN-LSTM) was trained using the Teacher's synthetic labels (on the 20-video set), its performance rose from 70.2% to 73.2% F1 score.

Notably, this approach allows the real-time model to close approximately half the performance gap between a data-starved model and one trained on the full 80-video manual dataset (which reaches 78.2% F1). This demonstrates that synthetic labels, while not perfect, provide enough workflow structure to significantly improve generalization.

Conclusion and Future Work

The teacher/student strategy offers a viable path for developing surgical AI in low-data regimes, effectively reducing the manual annotation burden by up to 75% while maintaining high performance. However, the method faces limitations in extreme scarcity; when only a single video is annotated, the teacher tends to overfit, producing noisy labels that hinder student training. Future research will explore pre-training via self-supervised proxy tasks and extending the framework to other complex surgical procedures.

Summary updated

Abstract

Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior. In laparoscopic procedures one particular algorithm needed for such systems is the identification of surgical phases, for which the current state of the art is a model based on a CNN-LSTM. A number of previous works using models of this kind have trained them in a fully supervised manner, requiring a fully annotated dataset. Instead, our work confronts the problem of learning surgical phase recognition in scenarios presenting scarce amounts of annotated data (under 25% of all available video recordings). We propose a teacher/student type of approach, where a strong predictor called the teacher, trained beforehand on a small dataset of ground truth-annotated videos, generates synthetic annotations for a larger dataset, which another model - the student - learns from. In our case, the teacher features a novel CNN-biLSTM-CRF architecture, designed for offline inference only. The student, on the other hand, is a CNN-LSTM capable of making real-time predictions. Results for various amounts of manually annotated videos demonstrate the superiority of the new CNN-biLSTM-CRF predictor as well as improved performance from the CNN-LSTM trained using synthetic labels generated for unannotated videos. For both offline and online surgical phase recognition with very few annotated recordings available, this new teacher/student strategy provides a valuable performance improvement by efficiently leveraging the unannotated data.
BibTeX
@misc{yu2018learning,
  author = {Tong Yu and Didier Mutter and Jacques Marescaux and Nicolas Padoy},
  title = {{Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition}},
  year = {2018},
  eprint = {1812.00033},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.00033},
}
Figure