TL;DR
TrackMem is a novel graph-based memory system designed for the "messy" reality of financial dialogue, where facts are frequently revised and stances evolve over time. By tracking information in "event time" rather than just "conversation time," it significantly outperforms existing memory architectures on complex, long-horizon investment queries, particularly when handling updated figures and shifting perspectives.
What Problem Does This Paper Address?
Most LLM memory systems (like RAG or MemGPT) are evaluated on social dialogues where facts are "append-only"—once something is said, it stays true. Financial dialogue breaks this assumption. In an investment context:
- Figures are revised: A price target set in January is superseded by a new one in March.
- Stances evolve: A user’s "bullish" outlook might shift to "neutral" based on a series of market events across multiple sessions.
- Constraints persist: A rule stated once (e.g., "never hold more than 5% in one stock") governs all future advice, even if it isn't mentioned again.
- Lenses matter: The same company might be discussed through different analytical lenses (valuation, regulatory risk, etc.) that need to remain distinct.
Standard retrieval systems often fail here because they grab the most "similar" or "recent" text, which might be an outdated figure, a hypothetical scenario, or a different speaker's opinion.
Methods
TrackMem moves away from flat logs and toward a structured, per-speaker property graph ($G_p$). The system categorizes information into four distinct node types, as illustrated in the design overview:
- Views: Summaries of a user's understanding of an entity (like a stock) under a specific topic or "lens."
- Anchors: Typed numeric values (e.g., price targets, yields) that maintain a versioned history.
- Claims: Immutable assertions from a single turn.
- Attributes: Stable, long-term user properties like risk tolerance or global portfolio constraints.
The core innovation is event-time tracking. Instead of just remembering when a turn occurred, TrackMem records the interval for which a fact is "in force." When a new session is ingested, a schema-constrained extraction process updates the graph. A "quantity resolver" decides if a new figure is a revision of an existing one or a separate data point.
Datasets
The authors introduce FinLoCoMo (Financial Long-Horizon Conversational Memory), a benchmark consisting of 15 conversations spanning 191–612 days. It includes 765 questions across two splits:
- Standard Split: Follows existing benchmarks (LoCoMo) with single-hop, multi-hop, and temporal questions.
- Modes Split: Specifically targets the four failure modes identified: Revision, Stance, Constraint, and Aspect.
Key Insights and Ideas
- Structure Beats Retrieval: The authors found that even when a flat RAG system is given the same "retrieval budget" (number of turns), TrackMem's structured approach is more accurate. This is because the graph explicitly tells the reader model which value is currently valid, whereas flat retrieval leaves the model to guess between conflicting mentions.
- The "Event Time" Advantage: By treating memory as a versioned history of state, the system avoids the "recency trap." It can distinguish between what was true at a specific point in the past versus what is true "now."
- Channel-Specific Retrieval: Different questions require different data. TrackMem uses a "Router" to decide which channels (Anchors, Trajectories, Views) to query based on the intent of the question.
Key Results
TrackMem consistently outperformed five major baselines (MemGPT, Mem0, A-Mem, Zep, and O-Mem) across both splits.
- Revision Mastery: On questions about revised figures, TrackMem reached 86.5% accuracy, compared to 73.4% for the strongest baseline (MemGPT).
- Superior Recall: As shown in Figure 1, TrackMem retrieves more "gold" (correct) evidence turns while maintaining a significantly smaller context window than a full-history approach.
- High Precision Abstention: When a question is unanswerable, TrackMem is better at "declining" to answer (91.5% precision) rather than hallucinating a wrong answer.
Limitations and Future Work
- LLM Dependency: The system relies on an LLM for the initial extraction and quantity resolution. If the model misidentifies an entity or merges two distinct figures during ingestion, that error persists in the memory.
- Domain and Language: The current evaluation is limited to English-language financial dialogues between two speakers. Its performance in multi-party dialogues or other domains (like legal or medical) remains untested.
- Ingestion Cost: Building the graph requires multiple model calls (extraction, resolution, synthesis), which is computationally more expensive than simple vector embedding used in standard RAG.
Summary updated