TL;DR
Researchers have developed ECHO-TD, a locally deployable dashboard that uses Retrieval-Augmented Generation (RAG) and graph analytics to analyze healthcare tele-mentoring transcripts. By combining a clinical LLM (MedGemma-4B) with strict metadata filtering and interaction graphs, the tool allows researchers to query clinical knowledge and quantify participation equity while ensuring total speaker anonymity and data privacy.
What Problem Does This Paper Address?
Project ECHO (Extension for Community Healthcare Outcomes) is a global movement that connects specialists with community providers via video sessions to discuss complex patient cases. These sessions generate a wealth of verbatim transcripts rich in clinical reasoning. However, analyzing this data at scale is a nightmare:
- Manual Review is Slow: Researchers can't efficiently sift through thousands of hours of dialogue.
- Privacy Risks: Sharing transcripts externally risks exposing participant identities.
- RAG Contamination: Standard RAG pipelines often "leak" context across different speakers or sessions, leading to confusing or incorrect answers.
- Lack of Interaction Data: Existing tools focus on what was said but ignore how people interacted—missing critical insights into who dominates the conversation and whether participation is equitable.
Methods
The authors built a modular pipeline, visualized in Figure 1, that runs entirely locally to maintain data security.
- Speaker-Aware Indexing: Transcripts are parsed into speaker-labeled chunks and stored in ChromaDB. Unlike standard RAG, they use a compound metadata filter (speaker + session ID) to ensure the LLM only "sees" relevant snippets, preventing cross-session contamination.
- Local LLM Generation: They utilize MedGemma-4B, a model fine-tuned for medical data, running locally on a CPU. This eliminates the need for cloud APIs and keeps sensitive clinical data on-premises.
- Interaction Graph Analytics: Using NetworkX, the tool builds directed graphs where nodes are speakers and edges represent turn-taking. They calculate metrics like Betweenness Centrality (who acts as a bridge) and PageRank (conversational influence).
- Reference-Free Evaluation: Since they lack "gold standard" answers for every possible query, they implemented a framework inspired by RAGAS. This includes six metrics like contextual precision and a custom "hallucination-risk" score based on token overlap.
Datasets
The tool was demonstrated using a corpus of 10 ECHO sessions:
- Five sessions from early 2016 and five from early 2020.
- Topics included chronic pain management, opioid prescribing protocols, and psychiatric comorbidities.
- The final indexed collection contained approximately 3,200 text chunks.
Key Insights and Ideas
- Isolation is Key for Multi-Speaker Data: The "aha moment" here is that standard vector search isn't enough for dialogue. Without strict metadata filtering, a query about "Dr. Smith's opinion" might accidentally pull in "Dr. Jones's response" from a different year.
- Graph Theory for Education: By applying PageRank and betweenness centrality to transcripts, the tool turns subjective observations about "who talks too much" into objective, quantitative data. This helps facilitators identify "non-contributors" (those in the bottom 15th percentile) who might need more encouragement.
- Privacy through Aliasing: Anonymization isn't just about scrubbing names; it's about consistent aliasing. By replacing names with labels like
hub_member1across the dashboard, graphs, and LLM responses, the tool maintains the narrative flow of the discussion without compromising identity.
Key Results
- Performance: After an initial 3.2-second load time, subsequent retrievals from the vector database take less than 0.5 seconds.
- Zero Leakage: The compound filtering achieved "zero cross-session chunk leakage," ensuring high analytical validity.
- Interactive Visualization: The dashboard provides physics-simulated graphs (Figure 4) that allow users to visually compare participation dynamics between two different sessions.
Limitations and Future Work
- Hardware Constraints: Running a 4-billion parameter model on a CPU is possible but introduces higher latency compared to GPU-accelerated environments, especially in "Longitudinal Mode" which queries multiple sessions.
- Dataset Scale: The current study only utilized 10 sessions. While the indexing is idempotent (allowing for easy scaling), the performance of the graph analytics and RAG retrieval on thousands of sessions remains to be seen.
- Heuristic Evaluation: While the reference-free metrics are a great proxy, they are still heuristics. Future work could involve expert clinicians validating the LLM's summaries to further refine the "hallucination-risk" scores.
Summary updated