Tri-Modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms

Diandian Guo, Manxi Lin, Jialun Pei, He Tang, Yueming Jin, Pheng-Ann Heng

Introduction

Automating the understanding of the Operating Room (OR) environment is a critical step toward surgical assistance and workflow optimization. This task requires robust Scene Graph Generation (SGG), which involves identifying subjects (e.g., surgeons, patients), objects (e.g., instruments, tables), and the complex, dynamic relationships between them (e.g., "Surgeon holding Instrument").

Existing SGG methods in medical contexts face two primary hurdles. First, temporal complexity: surgical actions are often subtle and evolve over long durations, making them difficult to capture with static or memory-limited models. Second, class imbalance: frequent states, such as a patient "lying on" a table, dominate datasets, while critical but brief actions like "cutting" or "sawing" are rare, leading to poor model sensitivity for high-stakes events.

Methodology

To address these challenges, we present TriTemp-OR, an end-to-end tri-modal framework that integrates 2D video, 3D point clouds, and biomedical language model knowledge to achieve a comprehensive understanding of surgical scenes.

The TriTemp-OR architecture integrates 2D video, 3D point clouds, and biomedical language embeddings to model temporal dynamics and semantic relationships.

Bi-modal Temporal Dynamics

TriTemp-OR moves beyond simple historical summarization by directly interacting with raw data streams:

  • ViewTemp (2D): This module utilizes scale-adaptive kernels to process multi-view video. It adjusts its receptive field based on the camera perspective—employing wider kernels for global room views and finer kernels for localized surgical views.
  • PointTemp (3D): This component employs 4D convolutions to track changes in the 3D geometry of the OR over time, ensuring the model maintains a consistent grasp of the physical space.

Knowledge Transfer via LLaVA-Med

To mitigate the effects of class imbalance, the framework leverages the "medical common sense" of LLaVA-Med, a Large Language Model (LLM) pre-trained on biomedical data. Through a process of knowledge distillation, the model generates rich text embeddings for surgical triplets (e.g., "A scene of a surgeon cutting a patient"). During training, visual features are aligned with these medical text embeddings, effectively teaching the model the semantic nuances of rare actions even when visual examples are scarce.

Experimental Results

The model was evaluated on the 4D-OR benchmark, which contains synchronized RGB-D data from 10 knee replacement surgeries. TriTemp-OR established a new state-of-the-art for the dataset, achieving an average Precision of 0.91 and an F1 score of 0.90.

A sample scene graph generated by TriTemp-OR, demonstrating its ability to accurately predict complex surgical relationships and actions.

Key Findings

  • Rare Class Recognition: The integration of LLaVA-Med significantly improved performance on infrequent but vital actions. Recognition of "Sawing" improved by 12% and "Cleaning" by 11% compared to previous benchmarks.
  • Semantic Clustering: Analysis revealed that biomedical LLMs provide superior semantic separation for surgical actions compared to general-purpose models like CLIP, which often struggle to distinguish between medically distinct activities.
  • Efficiency: Despite its tri-modal nature, TriTemp-OR is more parameter-efficient than its predecessors, utilizing 67.1M parameters compared to the 85.8M used by the LABRAD-OR model.

Conclusion and Future Work

TriTemp-OR demonstrates that the confluence of multi-modal temporal dynamics and specialized biomedical knowledge can significantly advance surgical scene understanding. While the results on knee replacements are promising, future research will focus on optimizing the pipeline for real-time deployment and testing the framework's generalizability across diverse surgical disciplines, such as abdominal or laparoscopic procedures.

Summary updated

BibTeX
@inproceedings{guo2024tri,
  author = {Diandian Guo and Manxi Lin and Jialun Pei and He Tang and Yueming Jin and Pheng-Ann Heng},
  title = {{Tri-Modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72089-5\_67},
}

Loading related content...

Figure