Abstract
Surgical video understanding requires more than simple object detection; it necessitates a deep comprehension of the relationships between instruments, anatomy, and safety protocols. We present ConceptNet, a temporal hypergraph neural network designed to integrate expert surgical knowledge directly into AI architectures. By representing surgical procedures as dynamic graphs of interacting concepts, ConceptNet achieves state-of-the-art performance in action triplet recognition, inflammation grading, and safety criteria assessment, while providing inherently interpretable outputs.
Introduction: The Need for Surgical Reasoning
While deep learning has advanced basic tasks like tool tracking, it often fails at "surgical reasoning"—the ability to understand how specific tool-tissue interactions contribute to procedural milestones. Surgeons perceive surgery as a structured web of relationships rather than isolated pixels. Existing models often lack this inductive bias, leading to "black box" behavior that struggles with generalization and interpretability. ConceptNet addresses this by injecting human-like prior knowledge into neural networks, enabling them to model the "why" and "how" of surgical procedures.
Methodology: ConceptNet Architecture
ConceptNet models the surgical process as a spatial-temporal hypergraph. Unlike standard graphs that only connect pairs of nodes, hypergraphs utilize hyperedges to connect multiple nodes, capturing complex N-ary relationships essential for clinical safety assessments.
The framework operates through three primary stages:
- Node Aggregation: Information from individual concept nodes (e.g., specific tools or tissues) is gathered to update the status of a relation (hyperedge).
- Hyperedge Propagation: Relational information is passed back to individual nodes to refine their states.
- Recurrent State Self-update: Each node and hyperedge maintains an internal memory via Long Short-Term Memory (LSTM) units, allowing the model to track the evolution of the surgery over time.
A visual backbone (such as ResNet50 or ViT) extracts features from video frames, which are then mapped onto this structured graph.
Knowledge Structures and Datasets
The effectiveness of ConceptNet relies on pre-acquired surgical knowledge structures that define the relationships between instruments, actions, and tissues.
The model was validated across three distinct tasks:
- Action Triplet Recognition (CholecT45): Identifying combinations of [instrument] - [action] - [tissue].
- Parkland Grading Scale (PGS): Assessing the severity of gallbladder inflammation on a 1-5 scale using the ParklandGradingScale200 dataset.
- Critical View of Safety (CVS): Evaluating whether specific anatomical landmarks are visible before proceeding, using the CVS100 dataset.
Key Results
ConceptNet consistently outperformed state-of-the-art baselines across all benchmarks:
- Action Triplets: Achieved a Mean Average Precision ($AP_{IVT}$) of 30.6%, a significant improvement over previous methods (27.2%).
- Inflammation Grading: Reached 67.5% accuracy on the Parkland scale, outperforming Temporal Shift Modules (62.1%).
- Safety Assessment: Achieved 67.1% accuracy in CVS detection, significantly higher than multitask learning baselines (~50-55%).
Discussion and Future Work
By constraining the model's search space with surgical textbooks' graph topologies, ConceptNet achieves higher efficiency and accuracy even with limited data. Its design ensures that outputs are interpretable, as every prediction corresponds to a defined clinical concept.
Future research will focus on automating the construction of these graph topologies to reduce manual design effort and extending the framework to more varied, less standardized surgical procedures beyond laparoscopic cholecystectomy.
Summary updated