Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms

Diandian Guo, Manxi Lin, Jialun Pei, He Tang, Yueming Jin, Pheng-Ann Heng

Introduction

Automating the understanding of the Operating Room (OR) environment is a critical step toward surgical assistance and workflow optimization. This task requires robust Scene Graph Generation (SGG), which involves identifying subjects (e.g., surgeons, patients), objects (e.g., instruments, tables), and the complex, dynamic relationships between them (e.g., "Surgeon holding Instrument").

Existing SGG methods in medical contexts face two primary hurdles. First, temporal complexity: surgical actions are often subtle and evolve over long durations, making them difficult to capture with static or memory-limited models. Second, class imbalance: frequent states, such as a patient "lying on" a table, dominate datasets, while critical but brief actions like "cutting" or "sawing" are rare, leading to poor model sensitivity for high-stakes events.

Methodology

To address these challenges, we present TriTemp-OR, an end-to-end tri-modal framework that integrates 2D video, 3D point clouds, and biomedical language model knowledge to achieve a comprehensive understanding of surgical scenes.

The TriTemp-OR architecture integrates 2D video, 3D point clouds, and biomedical language embeddings to model temporal dynamics and semantic relationships.

Bi-modal Temporal Dynamics

TriTemp-OR moves beyond simple historical summarization by directly interacting with raw data streams:

  • ViewTemp (2D): This module utilizes scale-adaptive kernels to process multi-view video. It adjusts its receptive field based on the camera perspective—employing wider kernels for global room views and finer kernels for localized surgical views.
  • PointTemp (3D): This component employs 4D convolutions to track changes in the 3D geometry of the OR over time, ensuring the model maintains a consistent grasp of the physical space.

Knowledge Transfer via LLaVA-Med

To mitigate the effects of class imbalance, the framework leverages the "medical common sense" of LLaVA-Med, a Large Language Model (LLM) pre-trained on biomedical data. Through a process of knowledge distillation, the model generates rich text embeddings for surgical triplets (e.g., "A scene of a surgeon cutting a patient"). During training, visual features are aligned with these medical text embeddings, effectively teaching the model the semantic nuances of rare actions even when visual examples are scarce.

Experimental Results

The model was evaluated on the 4D-OR benchmark, which contains synchronized RGB-D data from 10 knee replacement surgeries. TriTemp-OR established a new state-of-the-art for the dataset, achieving an average Precision of 0.91 and an F1 score of 0.90.

A sample scene graph generated by TriTemp-OR, demonstrating its ability to accurately predict complex surgical relationships and actions.

Key Findings

  • Rare Class Recognition: The integration of LLaVA-Med significantly improved performance on infrequent but vital actions. Recognition of "Sawing" improved by 12% and "Cleaning" by 11% compared to previous benchmarks.
  • Semantic Clustering: Analysis revealed that biomedical LLMs provide superior semantic separation for surgical actions compared to general-purpose models like CLIP, which often struggle to distinguish between medically distinct activities.
  • Efficiency: Despite its tri-modal nature, TriTemp-OR is more parameter-efficient than its predecessors, utilizing 67.1M parameters compared to the 85.8M used by the LABRAD-OR model.

Conclusion and Future Work

TriTemp-OR demonstrates that the confluence of multi-modal temporal dynamics and specialized biomedical knowledge can significantly advance surgical scene understanding. While the results on knee replacements are promising, future research will focus on optimizing the pipeline for real-time deployment and testing the framework's generalizability across diverse surgical disciplines, such as abdominal or laparoscopic procedures.

Summary updated

Abstract

A comprehensive understanding of surgical scenes allows for monitoring of the surgical process, reducing the occurrence of accidents and enhancing efficiency for medical professionals. Semantic modeling within operating rooms, as a scene graph generation (SGG) task, is challenging since it involves consecutive recognition of subtle surgical actions over prolonged periods. To address this challenge, we propose a Tri-modal (i.e., images, point clouds, and language) confluence with Temporal dynamics framework, termed TriTemp-OR. Diverging from previous approaches that integrated temporal information via memory graphs, our method embraces two advantages: 1) we directly exploit bi-modal temporal information from the video streaming for hierarchical feature interaction, and 2) the prior knowledge from Large Language Models (LLMs) is embedded to alleviate the class-imbalance problem in the operating theatre. Specifically, our model performs temporal interactions across 2D frames and 3D point clouds, including a scale-adaptive multi-view temporal interaction (ViewTemp) and a geometric-temporal point aggregation (PointTemp). Furthermore, we transfer knowledge from the biomedical LLM, LLaVA-Med, to deepen the comprehension of intraoperative relations. The proposed TriTemp-OR enables the aggregation of tri-modal features through relation-aware unification to predict relations so as to generate scene graphs. Experimental results on the 4D-OR benchmark demonstrate the superior performance of our model for long-term OR streaming.
BibTeX
@inproceedings{guo2024tri,
  author = {Diandian Guo and Manxi Lin and Jialun Pei and He Tang and Yueming Jin and Pheng-Ann Heng},
  title = {{Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72089-5\_67},
}
Figure