PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs

Jinyue Li, Yuci Liang, Qiankun Li, Xinheng Lyu, Jiayu Qian, Huabao Chen, Kun Wang, Zhigang Zeng, Anil Anthony Bharath, Yang Liu

In the world of computational pathology, looking at a slide is only half the battle. A human pathologist doesn't just see patterns; they constantly retrieve formal grading criteria, disease taxonomies, and clinical evidence from their long-term memory to make a diagnosis.

Current Multimodal Large Language Models (MLLMs) are great at describing images, but they often act like "parametric black boxes." They have a lot of "internalized" knowledge, but they lack a structured way to pull up specific medical rules when they need them. This leads to hallucinations or a failure to follow strict diagnostic standards.

PathMem, a new framework presented by Jinyue Li and colleagues, aims to fix this by giving AI a "cognition-aligned" memory system that mimics the human brain's transition from Long-Term Memory (LTM) to Working Memory (WM).

TL;DR

PathMem is a memory-centric framework for pathology AI that organizes expert medical knowledge into a structured Long-Term Memory (LTM). Using a novel "Memory Transformer," the model dynamically activates relevant knowledge into a "Working Memory" to ensure diagnostic reasoning is grounded in formal medical standards rather than just statistical patterns.

What Problem Does This Paper Address?

Pathology is a "knowledge-intensive" discipline. To diagnose a breast carcinoma, for example, you must apply the Nottingham grading system, which involves specific scores for tubule formation, nuclear pleomorphism, and mitotic count.

Existing pathology MLLMs struggle with this because:

  1. Static Retrieval: Current Retrieval-Augmented Generation (RAG) methods are often too rigid; they fetch information once and don't adapt as the model "thinks."
  2. Lack of Interpretability: It is hard to know if a model's diagnosis is based on actual medical criteria or just a lucky guess based on pixel patterns.
  3. Knowledge Integration: Models often fail to link visual "morphological evidence" (what the cells look like) with formal "diagnostic criteria" (the rules for what those cells mean).

Methods

The researchers built PathMem around two core innovations:

1. High-Quality LTM Construction

Instead of a generic database, the authors built a specialized Pathology Knowledge Graph (KG). They used deep semantic searches over PubMed abstracts and employed GPT-4o to extract "triples" (Subject-Relation-Object). To ensure the memory was reliable, they used a Probabilistic Multi-Evidence Fusion mechanism—essentially, if multiple papers agree on a medical fact, that fact gets a higher confidence weight in the model's memory.

2. The Memory Transformer

This is the "brain" of the operation. It manages the transition from LTM to Working Memory (WM) using a dual-mode process:

  • Static Activation: A quick lookup based on the similarity between the input text/image and the knowledge base.
  • Dynamic Activation: A more complex step where the model projects visual and textual data into a shared space to calculate "global relevance."
  • Adaptive Selection: Rather than just taking the "Top 5" facts, the model uses an adaptive strategy to decide exactly how many pieces of knowledge are needed for a specific case, filtering out noise and keeping the Working Memory focused.

Datasets

The model was trained and evaluated on a massive scale:

  • Internal Dataset: WSI-Bench, a morphology-aware benchmark containing nearly 180,000 VQA pairs across 30 cancer types.
  • External Benchmarks (Zero-shot): To prove the model actually "learned" how to reason (rather than just memorizing the test), they tested it on WSI-VQA, SlideBench-VQA (BCNB), and CPTAC-NSCLC without any additional fine-tuning.

Key Insights and Ideas

  • Cognitive Alignment Works: By mimicking the human LTM $\rightarrow$ WM pipeline, the model becomes significantly more consistent. It doesn't just "see" a tumor; it "remembers" the rules for that specific tumor type.
  • Dynamic Reweighting is Crucial: The paper shows that "Static" retrieval isn't enough. The "Dynamic" activation allows the model to realize that a specific visual cue (like a certain cell shape) should trigger a specific set of grading rules that might have been missed by a simple text search.
  • Interpretability by Design: Because the model explicitly moves "knowledge tokens" into its working memory, researchers can see exactly which medical rules the model used to reach its conclusion.

Key Results

PathMem achieved State-of-the-Art (SOTA) performance across the board:

  • Report Generation: On WSI-Bench, it improved WSI-Precision by 12.8% and WSI-Relevance by 10.1% compared to previous models like WSI-LLaVA.
  • Diagnostic Accuracy: In open-ended diagnosis tasks, it saw gains of 9.7% and 8.9% over prior benchmarks.
  • Zero-Shot Generalization: On the external BCNB dataset, PathMem hit a 0.918 score for tumor detection, significantly outperforming general-purpose models like GPT-4o, which struggled with the specialized pathology context.

Limitations and Future Work

While PathMem is a major step forward, the authors note a few areas for growth:

  • Knowledge Coverage: The LTM is currently built from PubMed abstracts. Expanding this to include full-text textbooks and more diverse clinical guidelines could make the "memory" even richer.
  • Computational Efficiency: Processing gigapixel Whole Slide Images (WSIs) alongside a large knowledge graph is computationally expensive. Future work will focus on making the WSI processing more efficient.
  • Clinical Validation: While the model performs well on benchmarks, the authors emphasize the need for further validation in real-world clinical workflows to ensure safety and reliability.

Paper Citation: Li, J., et al. "PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs." arXiv:2603.09943v1 [cs.AI], 2026.

Summary updated

Abstract

Computational pathology demands both visual pattern recognition and dynamic integration of structured domain knowledge, including taxonomy, grading criteria, and clinical evidence. In practice, diagnostic reasoning requires linking morphological evidence with formal diagnostic and grading criteria. Although multimodal large language models (MLLMs) demonstrate strong vision language reasoning capabilities, they lack explicit mechanisms for structured knowledge integration and interpretable memory control. As a result, existing models struggle to consistently incorporate pathology-specific diagnostic standards during reasoning. Inspired by the hierarchical memory process of human pathologists, we propose PathMem, a memory-centric multimodal framework for pathology MLLMs. PathMem organizes structured pathology knowledge as a long-term memory (LTM) and introduces a Memory Transformer that models the dynamic transition from LTM to working memory (WM) through multimodal memory activation and context-aware knowledge grounding, enabling context-aware memory refinement for downstream reasoning. PathMem achieves SOTA performance across benchmarks, improving WSI-Bench report generation (12.8% WSI-Precision, 10.1% WSI-Relevance) and open-ended diagnosis by 9.7% and 8.9% over prior WSI-based models.
BibTeX
@misc{li2026pathmem,
  author = {Jinyue Li and Yuci Liang and Qiankun Li and Xinheng Lyu and Jiayu Qian and Huabao Chen and Kun Wang and Zhigang Zeng and Anil Anthony Bharath and Yang Liu},
  title = {{PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs}},
  year = {2026},
  eprint = {2603.09943},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.09943},
}
Figure