Concept Graph Neural Networks for Surgical Video Understanding

Yutong Ban, Jennifer A. Eckhoff, Thomas M. Ward, Daniel A. Hashimoto, Ozanan R. Meireles, Daniela Rus, Guy Rosman

Abstract

Surgical video understanding requires more than simple object detection; it necessitates a deep comprehension of the relationships between instruments, anatomy, and safety protocols. We present ConceptNet, a temporal hypergraph neural network designed to integrate expert surgical knowledge directly into AI architectures. By representing surgical procedures as dynamic graphs of interacting concepts, ConceptNet achieves state-of-the-art performance in action triplet recognition, inflammation grading, and safety criteria assessment, while providing inherently interpretable outputs.

Introduction: The Need for Surgical Reasoning

While deep learning has advanced basic tasks like tool tracking, it often fails at "surgical reasoning"—the ability to understand how specific tool-tissue interactions contribute to procedural milestones. Surgeons perceive surgery as a structured web of relationships rather than isolated pixels. Existing models often lack this inductive bias, leading to "black box" behavior that struggles with generalization and interpretability. ConceptNet addresses this by injecting human-like prior knowledge into neural networks, enabling them to model the "why" and "how" of surgical procedures.

Methodology: ConceptNet Architecture

ConceptNet models the surgical process as a spatial-temporal hypergraph. Unlike standard graphs that only connect pairs of nodes, hypergraphs utilize hyperedges to connect multiple nodes, capturing complex N-ary relationships essential for clinical safety assessments.

Overview of the ConceptNet framework, illustrating node aggregation, hyperedge propagation, and recurrent state updates.

The framework operates through three primary stages:

  1. Node Aggregation: Information from individual concept nodes (e.g., specific tools or tissues) is gathered to update the status of a relation (hyperedge).
  2. Hyperedge Propagation: Relational information is passed back to individual nodes to refine their states.
  3. Recurrent State Self-update: Each node and hyperedge maintains an internal memory via Long Short-Term Memory (LSTM) units, allowing the model to track the evolution of the surgery over time.

A visual backbone (such as ResNet50 or ViT) extracts features from video frames, which are then mapped onto this structured graph.

Knowledge Structures and Datasets

The effectiveness of ConceptNet relies on pre-acquired surgical knowledge structures that define the relationships between instruments, actions, and tissues.

Knowledge structures for (a) action triplets, (b) Critical View of Safety (CVS), and (c) Parkland Grading Scale (PGS).

The model was validated across three distinct tasks:

  • Action Triplet Recognition (CholecT45): Identifying combinations of [instrument] - [action] - [tissue].
  • Parkland Grading Scale (PGS): Assessing the severity of gallbladder inflammation on a 1-5 scale using the ParklandGradingScale200 dataset.
  • Critical View of Safety (CVS): Evaluating whether specific anatomical landmarks are visible before proceeding, using the CVS100 dataset.

Key Results

ConceptNet consistently outperformed state-of-the-art baselines across all benchmarks:

  • Action Triplets: Achieved a Mean Average Precision ($AP_{IVT}$) of 30.6%, a significant improvement over previous methods (27.2%).
  • Inflammation Grading: Reached 67.5% accuracy on the Parkland scale, outperforming Temporal Shift Modules (62.1%).
  • Safety Assessment: Achieved 67.1% accuracy in CVS detection, significantly higher than multitask learning baselines (~50-55%).

Discussion and Future Work

By constraining the model's search space with surgical textbooks' graph topologies, ConceptNet achieves higher efficiency and accuracy even with limited data. Its design ensures that outputs are interpretable, as every prediction corresponds to a defined clinical concept.

Future research will focus on automating the construction of these graph topologies to reduce manual design effort and extending the framework to more varied, less standardized surgical procedures beyond laparoscopic cholecystectomy.

Summary updated

BibTeX
@article{ban2024concept,
  author = {Yutong Ban and Jennifer A. Eckhoff and Thomas M. Ward and Daniel A. Hashimoto and Ozanan R. Meireles and Daniela Rus and Guy Rosman},
  title = {{Concept Graph Neural Networks for Surgical Video Understanding}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2024},
  url = {https://openreview.net/forum?id=vx0G5ZAeg0},
}
Figure