TL;DR
This review surveys annotation tools for gastrointestinal polyp detection, a critical step in developing AI for capsule endoscopy. It highlights the need for user-friendly, efficient tools and identifies automated labeling as a key feature for improving accuracy and speed, though more comparative studies are needed.
What Problem Does This Paper Address?
Developing robust AI models for medical image analysis, particularly for detecting polyps in capsule endoscopy (CE) videos, requires large, well-annotated datasets. However, creating these datasets is a time-consuming and labor-intensive process. Existing annotation tools are often technically complex, making them difficult for clinicians (the primary users) to adopt. Furthermore, there's a lack of comprehensive overviews that specifically address the needs of gastrointestinal polyp annotation, leaving a gap in understanding which features are most crucial for this task.
Methods
The authors conducted a literature review to identify and describe existing annotation tools relevant to gastrointestinal polyp annotation. They focused on tools that are either domain-specific or general-purpose annotation systems that have been applied to medical imaging. The review synthesizes information from various sources, including published articles and personal correspondence, to evaluate these tools based on criteria such as efficiency, technical features, and availability. They propose a framework for evaluating these tools from a medical user's perspective, emphasizing practical aspects over purely technical ones.
Datasets
The paper does not introduce a new dataset. It discusses the need for large, annotated medical image databases for training deep neural networks (DNNs) and mentions the "Hyper-Kvasir dataset" as a source of frames with polyps used in comparative studies.
Key Insights and Ideas
- The Clinician's Perspective Matters: Annotation tools are often evaluated by technical experts, but their primary users are clinicians. This review attempts to bridge that gap by considering user-friendliness and practical utility.
- Efficiency is King (and Queen): Polyp annotation is inherently time-consuming. Tools that offer automated labeling or streamline the manual process significantly reduce the burden.
- Automated Labeling Shows Promise: Features like automated labeling and object tracking (e.g., in Encord) appear to be highly effective in improving annotation speed and potentially accuracy by following polyp trajectories.
- A Need for Standardization: While many tools support common formats, a lack of standardized evaluation metrics makes direct comparison difficult.
- "Something Is Missing": Beyond the annotation itself, the time spent finding, structuring, saving, and configuring data is often overlooked but contributes significantly to the overall workflow.
Key Results
The paper provides a comparative overview of several annotation tools, including CVAT, Encord (formerly Cord Vision), FastCAT, GTCreator, EIR System, and ImageLabeler (MATLAB).
- CVAT: A well-known, open-source tool, but found to be significantly slower than Encord in annotation speed (18.7/min vs. 121/min in one study).
- Encord: Presented as a novel annotation software with strong automated labeling features, showing a substantial improvement in annotation speed and task completion compared to CVAT. It claims to reduce expert workload.
- FastCAT: Designed for faster, less demanding annotation, it significantly reduces expert workload (by a factor of 20) and improves overall annotation speed, especially for non-experts. It was found to be more than twice as fast as CVAT for non-expert video annotation.
- GTCreator: Described as a flexible tool with structural advantages that lead to a significant reduction in mean total annotation time, partly due to features like GT revision and efficient navigation.
- Ratsnake: Offers manual and semi-automatic annotation, with a notable "snake" feature for efficient image annotation, particularly useful for polyps appearing in consecutive images.
- EIR System: Integrates manual and automatic annotation, with a semi-supervised tool that reduces time spent by automatically tracking ROIs, allowing non-experts to perform manual annotation.
- ImageLabeler (MATLAB): A feature-rich tool for image annotation, offering various manual and automated options, but its application to polyp annotation is based on reviewer experience rather than published studies.
Limitations and Future Work
- Limited Comparative Data: The authors note a lack of comprehensive comparative studies, especially regarding automated labeling and its impact on efficiency and accuracy across different tools.
- Inadequate Research Material: For polyp annotation specifically, the available research material is described as inadequate, often lacking preferable characteristics for medical staff due to overly technical details.
- Need for User-Centric Evaluation: Future work should focus on developing more practical evaluations of annotation tools from a medical user's perspective.
- Investigating True Positives: Further investigation is needed to understand the differences in true positive polyp annotations between manual and automated labeling methods.
- Time Consumption Beyond Annotation: Future research could explore the time consumption associated with data management (finding, structuring, saving, configuring) in annotation workflows.
Summary updated