TL;DR
This paper introduces AdaptOR, a novel unsupervised domain adaptation method to enable accurate clinician pose estimation and instance segmentation in operating rooms (ORs) using readily available in-the-wild images. AdaptOR tackles the significant domain shift and privacy concerns in ORs by leveraging geometric constraints on data augmentations and disentangled feature normalization, achieving strong performance even on low-resolution, privacy-preserving OR images.
What Problem Does This Paper Address?
The operating room (OR) is a complex environment where understanding clinician actions and spatial layout is crucial for developing advanced AI-powered support systems. This requires precise localization of clinicians through pose estimation (identifying key body points) and instance segmentation (outlining each individual). However, applying standard computer vision models to OR data is challenging due to:
- Domain Shift: OR images look vastly different from typical datasets used for training (e.g., natural images).
- Data Scarcity & Privacy: Collecting and annotating OR data is difficult and expensive due to privacy concerns, especially for pixel-level segmentation. To mitigate privacy issues, OR videos are often recorded at low resolutions, further degrading image quality and making analysis harder.
Existing fully supervised methods require extensive manual annotations on OR data, which is impractical. Unsupervised Domain Adaptation (UDA) offers a promising solution by adapting models trained on labeled "source" domains to unlabeled "target" domains. This paper specifically addresses the UDA challenge for the single, challenging class of "person" within the unique context of the OR, particularly focusing on low-resolution images.
Methods
AdaptOR builds upon a Mask R-CNN backbone, a standard for joint pose estimation and instance segmentation. The core innovations are:
- Disentangled Feature Normalization (DFN): To handle the distinct statistical properties of source and target domains, DFN replaces standard normalization layers (like Group Normalization) with separate layers for source and target data. This allows the model to learn domain-specific feature representations while still enabling joint training.
- Self-Training with Geometric Constraints: The method employs a self-training framework where a model generates "pseudo-labels" for unlabeled target domain images. Crucially, these pseudo-labels are generated by exploiting transformation equivariance. Instead of assuming invariance (labels don't change with transformation), they leverage the fact that spatial localization labels do change predictably with transformations (like flipping or resizing). This allows for generating more accurate pseudo-labels by applying geometric constraints derived from these transformations.
- Advanced Data Augmentation: To specifically address low-resolution OR images and privacy concerns, a "strong-resize" augmentation is introduced. This involves downsampling and then upsampling images with scaling factors up to 12x, creating highly blurred images that mimic privacy-preserving recordings. This augmentation, along with others like random-flip and random-resize, is used to create "weak" and "strong" augmented versions of the unlabeled target images.
- Mean-Teacher Framework: To stabilize the self-training process, a mean-teacher approach is used. A "teacher" model (an EMA of the "student" model) generates pseudo-labels on weakly augmented images, which are then used to train the "student" model on strongly augmented images.
Datasets
- Source Domain: COCO (Lin et al., 2014) - a large-scale dataset of natural images with annotations for bounding boxes, keypoints, and segmentation masks.
- Target Domains:
- MVOR+ (Srivastav et al., 2018, 2020) - OR images from real surgical interventions, extended with COCO-format full-body keypoints.
- TUM-OR-test (Belagiannis et al., 2016) - OR images from simulated surgical activities. This dataset was re-annotated by the authors for this work.
The paper also evaluates the method as a Semi-Supervised Learning (SSL) approach on the COCO dataset.
Key Insights and Ideas
- Geometric Equivariance is Key for Localization: Unlike classification tasks where invariance is desired, spatial localization tasks (like pose estimation) benefit from understanding how labels transform with image changes. This insight is central to generating accurate pseudo-labels.
- Low-Resolution Images as a Privacy Tool: The paper highlights that downsampling images to low resolutions is a practical approach for privacy in ORs. Their method is specifically designed to handle the significant degradation caused by this.
- Disentangled Normalization for Domain Adaptation: Simply aligning features isn't always enough. Explicitly normalizing source and target domain features separately (DFN) helps the model learn domain-specific characteristics while still adapting effectively.
- Self-Training Can Be Powerful Even for Complex Tasks: By combining self-training with carefully designed augmentations and geometric constraints, the authors show it's possible to achieve good performance on challenging tasks like joint pose estimation and instance segmentation without target domain labels.
- Generality as SSL: The proposed UDA framework can be repurposed as a strong SSL method, achieving impressive results with very limited labeled data.
Key Results
- AdaptOR outperforms baselines: On both MVOR+ and TUM-OR-test datasets, AdaptOR significantly outperforms strong baselines (KM-PL, KM-DDS, KM-ORPose) across various downsampling scales (1x to 12x) and target resolutions.
- Strong performance on low-resolution images: AdaptOR shows substantial improvements on low-resolution images (8x, 10x, 12x downsampling), achieving up to ~57% AP for bounding boxes and ~63% AP for keypoints on TUM-OR-test at 12x downsampling.
- Effective privacy preservation: The method demonstrates its utility for privacy-preserving OR environments by maintaining good performance even at 12x downsampling.
- SSL performance: As an SSL method on COCO, AdaptOR-SSL achieves comparable results to a fully supervised model with as little as 1% of labeled data (e.g., 42.57 APbb person with 1% labels vs. 52.0 APbb person with 100% labels).
- Ablation studies confirm component importance: DFN and the strong-resize augmentation were shown to be critical for performance, especially on low-resolution data.
Limitations and Future Work
- Domain Gap: While AdaptOR significantly reduces the domain gap, a performance gap still exists compared to fully supervised methods on the source domain, particularly for keypoint estimation.
- Teacher Model Complexity: The paper notes that using a complex multi-stage teacher model (as in KM-ORPose) can sometimes lead to a slight decrease in keypoint accuracy, suggesting that simpler, model-agnostic approaches might be more robust.
- Future Work: The authors suggest that their work provides valuable baselines for joint person pose estimation and instance segmentation on large-scale public datasets, potentially fostering further research in SSL. The generality of their approach as an SSL method also opens avenues for exploring its use in other domains with limited labeled data.
Summary updated