EndoAgent: A Memory-Guided Reflective Agent for Intelligent Endoscopic Vision-to-Decision Reasoning

Yi Tang, Kaini Wang, Yang Chen, Guangquan ZhouarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces EndoCogniAgent, a novel closed-loop reasoning framework for endoscopic diagnosis that addresses the critical issues of hallucinated evidence and error accumulation in current AI systems. By incorporating a self-consistency validation mechanism and an episodic memory module, EndoCogniAgent iteratively refines its diagnostic state, leading to more reliable and clinically acceptable diagnoses.

What Problem Does This Paper Address?

Endoscopic diagnosis is a complex, iterative process where clinicians gather and verify visual evidence to reach a conclusion. Existing AI systems struggle to replicate this process effectively because they often weakly couple fine-grained evidence acquisition with multi-step reasoning. This leads to two major failure modes: "hallucinated evidence" (where AI generates plausible but incorrect findings based on weak cues) and "error accumulation" (where initial mistakes propagate and influence subsequent reasoning). These issues undermine the reliability and trustworthiness of AI in clinical settings, especially in endoscopy where subtle visual details are crucial for accurate diagnosis.

Methods

EndoCogniAgent employs a closed-loop, agentic framework designed to mimic the iterative nature of clinical diagnosis. The core components include:

  • Structured Diagnostic Planning: A Central Decision Unit (CDU), implemented using an LLM, plans the next evidence acquisition step based on the current diagnostic state. This moves away from a single-shot prediction to a sequential decision process.
  • Clinical Evidence Extraction: A Perceptual Expert Toolset (PET) comprises specialized models (e.g., for classification, detection, segmentation) that extract specific clinical evidence from the endoscopic image based on the planner's action.
  • Internal Self-Consistency Validation (ISCV): This is the key innovation. Before any newly acquired evidence is integrated into the diagnostic state, it undergoes validation on two fronts:
  • Knowledge Consistency: Checks if the observation is grounded in visible image cues (e.g., morphology, vascular patterns).
  • Temporal Consistency: Checks if the observation is coherent with previously established findings in the diagnostic trajectory.
  • If validation fails, corrective feedback is generated to guide further verification or refinement.
  • Episodic Memory Module (EMM): Stores the validated diagnostic state, including actions, extracted evidence, and validation outcomes. This memory is crucial for conditioning subsequent planning and ensuring long-range consistency.
  • Validation-Guided State Update: The diagnostic state is updated only after successful validation, preventing unverified or inconsistent information from influencing the reasoning process.

Datasets

The paper introduces EndoAgentBench, a new benchmark designed for evaluating endoscopic diagnostic agents across a comprehensive workflow. It comprises:

  • 6,132 question-answer pairs derived from 11 endoscopic datasets.
  • These datasets include 10 public datasets and 1 private dataset.
  • The benchmark covers 5 diagnostic tasks: Lesion Classification, Lesion Quantification, Visual Grounding, Image Captioning, and Report Generation.
  • Samples are categorized into 4 lesion types: normal, polyp, adenoma, and cancer.

Key Insights and Ideas

  • The critical role of validation: Simply integrating more tools or data isn't enough. Explicit, multi-dimensional validation of evidence before it updates the diagnostic state is paramount to prevent errors.
  • Iterative refinement is key: Diagnosis is not a one-shot process. Allowing the agent to iteratively refine its understanding through planning, extraction, validation, and memory updates leads to significantly better outcomes.
  • Decomposition improves performance: Breaking down the complex diagnostic task into distinct stages (planning, extraction, validation) and leveraging specialized "expert tools" allows for more robust and accurate reasoning than monolithic models.
  • End-to-end workflow evaluation is necessary: Traditional benchmarks often focus on isolated tasks. EndoAgentBench's workflow-oriented approach is crucial for assessing how well agents integrate perception and reasoning in a clinically relevant manner.

Key Results

  • Perception Tasks: EndoCogniAgent achieved an overall accuracy of 85.23% on visual perception tasks, significantly outperforming the strongest general-purpose MLLM baseline (Gemini 2.5 Pro at 69.82%) by 15.41%.
  • Reasoning Tasks: The agent achieved a Clinical Acceptance Rate (CAR) of 71.13% on high-level diagnostic reasoning tasks.
  • Ablation Studies:
    • The "Agentic Core" (PET + CDU) significantly improved visual grounding.
  • ISCV was critical for improving both perception and reasoning, boosting MRG CAR from 46.81% to 70.83%.
  • The EMM further optimized performance, leading to the final state-of-the-art results.
  • Tool Orchestration: The CDU adaptively orchestrates tools based on task complexity, showing a clear shift from single-tool reliance for simple tasks to multi-expert coordination for complex reasoning like report generation.

Limitations and Future Work

  • Latency: The current framework relies on API-based LLMs, which introduces latency that might be prohibitive for real-time deployment.
  • Static Images: The benchmark and evaluation were conducted on still images, not video sequences.
  • Clinical Validation: Prospective clinical evaluation has not yet been performed.

Future work will focus on:

  • Extending the framework to temporal video reasoning.
  • Optimizing for real-time performance.
  • Conducting thorough validation in real clinical workflows.

Summary updated

Abstract

Endoscopic diagnosis is an iterative process in which clinicians progressively acquire, compare, and verify local visual evidence before reaching a conclusion. Current AI systems do not adequately support this process because fine-grained evidence acquisition and multi-step reasoning remain weakly coupled. This gives rise to two failure modes, hallucinated evidence and uncorrected error accumulation, that undermine diagnostic reliability. We propose EndoCogniAgent, a closed-loop agentic framework that formulates endoscopic diagnosis as a controlled state update process. At each reasoning round, a central planner selects the next evidence acquisition action, specialized expert tools extract the corresponding observation, and a self-consistency validation mechanism examines the observation along two dimensions, knowledge consistency against the input image and temporal consistency with prior validated findings, before updating the diagnostic state. Validated observations are admitted into the evolving state to condition subsequent planning, while insufficiently supported findings are retained with corrective feedback that redirects the planner toward additional verification. We further introduce EndoAgentBench, a workflow-oriented benchmark comprising 6,132 question-answer pairs from 11 endoscopic datasets, designed to evaluate diagnostic agents across a comprehensive diagnostic chain, from fine-grained visual perception to high-level diagnostic reasoning. Experiments show that EndoCogniAgent achieves 85.23\% average accuracy on perception tasks and 71.13\% clinical acceptance rate on reasoning tasks, with ablation analysis confirming that self-consistency validation and episodic state maintenance are individually critical to these gains.
BibTeX
@misc{tang2025endoagent,
  author = {Yi Tang and Kaini Wang and Yang Chen and Guangquan Zhou},
  title = {{EndoAgent: A Memory-Guided Reflective Agent for Intelligent Endoscopic Vision-to-Decision Reasoning}},
  year = {2025},
  eprint = {2508.07292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.07292},
}
Figure