A Tool for Anonymized Clinical Knowledge Retrieval and Speaker Interaction Analysis of Healthcare Tele-mentoring Sessions

Parthaw Goswami, Erfan Zareei, Amy Vogelsmeier, Dima Dandachi, Ben Casterline, Mirna Becevic, Praveen RaoarXiv 2026 · arXiv

TL;DR

Researchers have developed ECHO-TD, a locally deployable dashboard that uses Retrieval-Augmented Generation (RAG) and graph analytics to analyze healthcare tele-mentoring transcripts. By combining a clinical LLM (MedGemma-4B) with strict metadata filtering and interaction graphs, the tool allows researchers to query clinical knowledge and quantify participation equity while ensuring total speaker anonymity and data privacy.

What Problem Does This Paper Address?

Project ECHO (Extension for Community Healthcare Outcomes) is a global movement that connects specialists with community providers via video sessions to discuss complex patient cases. These sessions generate a wealth of verbatim transcripts rich in clinical reasoning. However, analyzing this data at scale is a nightmare:

  1. Manual Review is Slow: Researchers can't efficiently sift through thousands of hours of dialogue.
  2. Privacy Risks: Sharing transcripts externally risks exposing participant identities.
  3. RAG Contamination: Standard RAG pipelines often "leak" context across different speakers or sessions, leading to confusing or incorrect answers.
  4. Lack of Interaction Data: Existing tools focus on what was said but ignore how people interacted—missing critical insights into who dominates the conversation and whether participation is equitable.

Methods

The authors built a modular pipeline, visualized in Figure 1, that runs entirely locally to maintain data security.

Fig. 1. System architecture of RAG-based ECHO telehealth sessions transcript analysis dashboard.
  • Speaker-Aware Indexing: Transcripts are parsed into speaker-labeled chunks and stored in ChromaDB. Unlike standard RAG, they use a compound metadata filter (speaker + session ID) to ensure the LLM only "sees" relevant snippets, preventing cross-session contamination.
  • Local LLM Generation: They utilize MedGemma-4B, a model fine-tuned for medical data, running locally on a CPU. This eliminates the need for cloud APIs and keeps sensitive clinical data on-premises.
  • Interaction Graph Analytics: Using NetworkX, the tool builds directed graphs where nodes are speakers and edges represent turn-taking. They calculate metrics like Betweenness Centrality (who acts as a bridge) and PageRank (conversational influence).
  • Reference-Free Evaluation: Since they lack "gold standard" answers for every possible query, they implemented a framework inspired by RAGAS. This includes six metrics like contextual precision and a custom "hallucination-risk" score based on token overlap.

Datasets

The tool was demonstrated using a corpus of 10 ECHO sessions:

  • Five sessions from early 2016 and five from early 2020.
  • Topics included chronic pain management, opioid prescribing protocols, and psychiatric comorbidities.
  • The final indexed collection contained approximately 3,200 text chunks.

Key Insights and Ideas

  • Isolation is Key for Multi-Speaker Data: The "aha moment" here is that standard vector search isn't enough for dialogue. Without strict metadata filtering, a query about "Dr. Smith's opinion" might accidentally pull in "Dr. Jones's response" from a different year.
  • Graph Theory for Education: By applying PageRank and betweenness centrality to transcripts, the tool turns subjective observations about "who talks too much" into objective, quantitative data. This helps facilitators identify "non-contributors" (those in the bottom 15th percentile) who might need more encouragement.
  • Privacy through Aliasing: Anonymization isn't just about scrubbing names; it's about consistent aliasing. By replacing names with labels like hub_member1 across the dashboard, graphs, and LLM responses, the tool maintains the narrative flow of the discussion without compromising identity.
Fig. 2. Transcript analysis dashboard showing speaker names are replaced by hub/non-hub members.

Key Results

  • Performance: After an initial 3.2-second load time, subsequent retrievals from the vector database take less than 0.5 seconds.
  • Zero Leakage: The compound filtering achieved "zero cross-session chunk leakage," ensuring high analytical validity.
  • Interactive Visualization: The dashboard provides physics-simulated graphs (Figure 4) that allow users to visually compare participation dynamics between two different sessions.
Interaction graph showing speaker influence and turn-taking

Limitations and Future Work

  • Hardware Constraints: Running a 4-billion parameter model on a CPU is possible but introduces higher latency compared to GPU-accelerated environments, especially in "Longitudinal Mode" which queries multiple sessions.
  • Dataset Scale: The current study only utilized 10 sessions. While the indexing is idempotent (allowing for easy scaling), the performance of the graph analytics and RAG retrieval on thousands of sessions remains to be seen.
  • Heuristic Evaluation: While the reference-free metrics are a great proxy, they are still heuristics. Future work could involve expert clinicians validating the LLM's summaries to further refine the "hallucination-risk" scores.

Summary updated

Abstract

We present a tool for anonymized retrieval of clinical knowledge and analysis of multi-speaker interactions in tele-mentoring sessions of Project ECHO (Extension for Community Healthcare Outcomes). Our tool's novel design includes: (a) a compound metadata filter that enforces strict speaker- and session-level isolation during retrieval, preventing cross-session context contamination; (b) MedGemma-4B as a locally-hosted language model for clinically grounded answer generation across three query modes; (c) interaction graphs to compute betweenness centrality, PageRank influence, and turn-taking patterns to quantify participation equity; and (d) a consistent alias-based anonymization scheme that prevents real participant names from appearing in any rendered output. During the demo, a healthcare provider (user) will interactively query multi-year ECHO sessions, explore participation dynamics through interaction graphs, compare sessions side-by-side, and assess answer quality via six reference-free RAG metrics, all without any cloud API dependency.
BibTeX
@misc{goswami2026tool,
  author = {Parthaw Goswami and Erfan Zareei and Amy Vogelsmeier and Dima Dandachi and Ben Casterline and Mirna Becevic and Praveen Rao},
  title = {{A Tool for Anonymized Clinical Knowledge Retrieval and Speaker Interaction Analysis of Healthcare Tele-mentoring Sessions}},
  year = {2026},
  eprint = {2611.10922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10922},
}
Figure