TL;DR
This paper introduces EndoCogniAgent, a novel closed-loop reasoning framework for endoscopic diagnosis that addresses the critical issues of hallucinated evidence and error accumulation in current AI systems. By incorporating a self-consistency validation mechanism and an episodic memory module, EndoCogniAgent iteratively refines its diagnostic state, leading to more reliable and clinically acceptable diagnoses.
What Problem Does This Paper Address?
Endoscopic diagnosis is a complex, iterative process where clinicians gather and verify visual evidence to reach a conclusion. Existing AI systems struggle to replicate this process effectively because they often weakly couple fine-grained evidence acquisition with multi-step reasoning. This leads to two major failure modes: "hallucinated evidence" (where AI generates plausible but incorrect findings based on weak cues) and "error accumulation" (where initial mistakes propagate and influence subsequent reasoning). These issues undermine the reliability and trustworthiness of AI in clinical settings, especially in endoscopy where subtle visual details are crucial for accurate diagnosis.
Methods
EndoCogniAgent employs a closed-loop, agentic framework designed to mimic the iterative nature of clinical diagnosis. The core components include:
- Structured Diagnostic Planning: A Central Decision Unit (CDU), implemented using an LLM, plans the next evidence acquisition step based on the current diagnostic state. This moves away from a single-shot prediction to a sequential decision process.
- Clinical Evidence Extraction: A Perceptual Expert Toolset (PET) comprises specialized models (e.g., for classification, detection, segmentation) that extract specific clinical evidence from the endoscopic image based on the planner's action.
- Internal Self-Consistency Validation (ISCV): This is the key innovation. Before any newly acquired evidence is integrated into the diagnostic state, it undergoes validation on two fronts:
- Knowledge Consistency: Checks if the observation is grounded in visible image cues (e.g., morphology, vascular patterns).
- Temporal Consistency: Checks if the observation is coherent with previously established findings in the diagnostic trajectory.
- If validation fails, corrective feedback is generated to guide further verification or refinement.
- Episodic Memory Module (EMM): Stores the validated diagnostic state, including actions, extracted evidence, and validation outcomes. This memory is crucial for conditioning subsequent planning and ensuring long-range consistency.
- Validation-Guided State Update: The diagnostic state is updated only after successful validation, preventing unverified or inconsistent information from influencing the reasoning process.
Datasets
The paper introduces EndoAgentBench, a new benchmark designed for evaluating endoscopic diagnostic agents across a comprehensive workflow. It comprises:
- 6,132 question-answer pairs derived from 11 endoscopic datasets.
- These datasets include 10 public datasets and 1 private dataset.
- The benchmark covers 5 diagnostic tasks: Lesion Classification, Lesion Quantification, Visual Grounding, Image Captioning, and Report Generation.
- Samples are categorized into 4 lesion types: normal, polyp, adenoma, and cancer.
Key Insights and Ideas
- The critical role of validation: Simply integrating more tools or data isn't enough. Explicit, multi-dimensional validation of evidence before it updates the diagnostic state is paramount to prevent errors.
- Iterative refinement is key: Diagnosis is not a one-shot process. Allowing the agent to iteratively refine its understanding through planning, extraction, validation, and memory updates leads to significantly better outcomes.
- Decomposition improves performance: Breaking down the complex diagnostic task into distinct stages (planning, extraction, validation) and leveraging specialized "expert tools" allows for more robust and accurate reasoning than monolithic models.
- End-to-end workflow evaluation is necessary: Traditional benchmarks often focus on isolated tasks. EndoAgentBench's workflow-oriented approach is crucial for assessing how well agents integrate perception and reasoning in a clinically relevant manner.
Key Results
- Perception Tasks: EndoCogniAgent achieved an overall accuracy of 85.23% on visual perception tasks, significantly outperforming the strongest general-purpose MLLM baseline (Gemini 2.5 Pro at 69.82%) by 15.41%.
- Reasoning Tasks: The agent achieved a Clinical Acceptance Rate (CAR) of 71.13% on high-level diagnostic reasoning tasks.
- Ablation Studies:
- The "Agentic Core" (PET + CDU) significantly improved visual grounding.
- ISCV was critical for improving both perception and reasoning, boosting MRG CAR from 46.81% to 70.83%.
- The EMM further optimized performance, leading to the final state-of-the-art results.
- Tool Orchestration: The CDU adaptively orchestrates tools based on task complexity, showing a clear shift from single-tool reliance for simple tasks to multi-expert coordination for complex reasoning like report generation.
Limitations and Future Work
- Latency: The current framework relies on API-based LLMs, which introduces latency that might be prohibitive for real-time deployment.
- Static Images: The benchmark and evaluation were conducted on still images, not video sequences.
- Clinical Validation: Prospective clinical evaluation has not yet been performed.
Future work will focus on:
- Extending the framework to temporal video reasoning.
- Optimizing for real-time performance.
- Conducting thorough validation in real clinical workflows.
Summary updated