Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the OR

Vinkle Srivastav, Afshin Gangi, Nicolas Padoy

Introduction

Computer vision applications in the operating room (OR) offer significant benefits, such as monitoring radiation exposure and analyzing surgical workflows. However, deploying these systems is hindered by two primary challenges: privacy concerns and the high cost of data annotation. High-resolution video recording is often restricted due to privacy regulations for patients and staff. While low-resolution recording is a viable alternative, standard models trained on general datasets (e.g., COCO) often fail in the specialized, blurry environment of an OR. Furthermore, annotating OR data is labor-intensive, often requiring up to 90 minutes per image.

The AdaptOR Framework

To address these challenges, we present AdaptOR, an unsupervised domain adaptation (UDA) framework designed to transfer knowledge from labeled "in-the-wild" datasets to unlabeled OR environments. Built upon a Mask R-CNN backbone, AdaptOR introduces several key innovations:

Overview of the AdaptOR framework, illustrating the domain shift from labeled source data (e.g., COCO) to unlabeled target OR data and the resulting performance improvements.

Disentangled Feature Normalization (DFN)

Standard models typically use uniform normalization parameters. AdaptOR employs separate Group Normalization layers for the source and target domains. This "disentanglement" allows the model to learn domain-specific features independently, preventing statistical interference between disparate environments like a public park and a sterile OR.

Transformation Equivariant Constraints

The framework utilizes a Mean-Teacher architecture. A "Teacher" model processes weakly augmented OR images to generate pseudo-labels, which a "Student" model then attempts to predict on strongly augmented versions of the same images. To ensure pseudo-label reliability, geometric constraints (such as chirality) are applied: if an image is flipped, the corresponding keypoints must also flip consistently.

Privacy-Preserving Training

AdaptOR is specifically designed for robustness against heavy downsampling. By incorporating "strong-resize" augmentations that downsample images by up to 12x during training, the model learns to extract meaningful features from blurred inputs that effectively anonymize individuals.

Experimental Setup

The framework was evaluated using three datasets:

  • COCO: The source domain, providing 57k labeled images of people in everyday settings.
  • MVOR+: A target domain dataset featuring real surgical interventions.
  • TUM-OR-test: Simulated clinical activities re-annotated with 2D keypoints for evaluation.

Results and Discussion

AdaptOR demonstrates significant improvements over source-only models and existing UDA baselines.

Qualitative comparison showing the impact of adaptation on pose estimation at 1x and 12x resolutions. Note the significant improvement in keypoint accuracy after adaptation, especially in the low-resolution (12x) scenario.

Robustness to Low Resolution

On 12x downsampled images—where human identification becomes difficult—AdaptOR achieved a bounding box Average Precision (AP) of 57.3% on MVOR+, nearly doubling the 29.6% achieved by standard non-adapted models.

Data Efficiency

In semi-supervised tests on the COCO dataset, AdaptOR achieved performance with only 1% of labeled data that was comparable to models trained on the full dataset, highlighting its efficiency in low-label regimes.

Qualitative Performance

Visual analysis confirms that the model accurately identifies clinicians even under heavy occlusion by surgical equipment or other personnel, maintaining consistent tracking across varying resolutions.

Conclusion

AdaptOR effectively bridges the gap between general-purpose vision models and the specialized requirements of the operating room. By leveraging geometric constraints and disentangled normalization, it enables accurate clinician tracking in privacy-sensitive, low-resolution environments without requiring manual OR labels. Future research will explore test-time adaptation and the integration of temporal video data to further enhance tracking stability.

Summary updated

BibTeX
@article{srivastav2022unsupervised,
  author = {Vinkle Srivastav and Afshin Gangi and Nicolas Padoy},
  title = {{Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the OR}},
  journal = {Medical Image Analysis},
  year = {2022},
  doi = {10.1016/j.media.2022.102525},
}
Figure