TL;DR
Scalaris is a novel memory architecture for enterprise AI assistants that breaks the traditional trade-off between retrieval accuracy and token efficiency. By treating user interactions (relational/long-term) and assistant responses (contextual/short-term) as fundamentally different data types, it achieves state-of-the-art accuracy while using 10x fewer retrieved tokens than leading systems like Hindsight.
What Problem Does This Paper Address?
Building a conversational AI that "remembers" you across months of interactions is hard. At enterprise scale, you face a brutal accuracy-efficiency trade-off:
- High Accuracy, High Cost: Systems like Hindsight retrieve massive chunks of raw history to ensure the model has all the facts. This leads to "context bloat," driving up inference costs and latency.
- Low Cost, Low Accuracy: Systems like Zep use aggressive summarization or lightweight graphs, but often miss the specific nuances needed to answer complex, multi-session questions.
Furthermore, most existing systems treat every message the same. The authors argue this is a mistake: user messages contain evolving facts about preferences, while assistant messages are often verbose, contextual, and become "stale" quickly (especially in fast-moving fields like finance).
Methods
Scalaris introduces a source-aware memory organization that splits the ingestion and storage pipeline based on who is speaking:
- User Interactions (Long-term): These are processed into a Temporal Knowledge Graph (built on an enhanced version of Graphiti). It extracts atomic facts and entities, tracking their validity over time (e.g., "John prefers growth stocks" might change to "John prefers value stocks").
- Assistant Responses (Short-term): Instead of storing full responses, Scalaris breaks them into atomic memory units. These are filtered for "low-value" content (like greetings) and stored in a Vector Store. This allows the system to retrieve only the specific informative snippet of a past answer rather than the whole verbose transcript.
- The Retrieval Pipeline: When a new query comes in, Scalaris first performs Query Reformulation to resolve pronouns (e.g., changing "What about that one?" to "What about the Toyota Corolla?"). It then executes a hybrid search across the graph and vector stores using semantic, lexical, and graph-traversal techniques.
Datasets
The authors evaluated Scalaris on two primary long-term memory benchmarks and one real-world production set:
- LongMemEval-S: 500 questions testing long-term recall and reasoning.
- LoCoMo: 1,986 questions across multi-session conversations.
- Internal Production Data: 1,200 real-world interactions from Motilal Oswal’s proprietary financial assistants to validate performance in the wild.
Key Insights and Ideas
- Functional Asymmetry: The core "aha moment" is that user inputs and assistant outputs serve different roles. User inputs are relational (building a profile), while assistant outputs are informational (providing data points). Tailoring the storage medium to these roles is what enables the 10x efficiency gain.
- Atomic Assistant Memory: By fragmenting assistant responses into self-contained units, the system avoids the "Lost in the Middle" problem where LLMs struggle with overly long contexts.
- Temporal Awareness: The use of a bi-temporal model (tracking when a fact was learned vs. when it is valid) prevents the assistant from retrieving "stale" information, which is critical for enterprise applications like finance.
Key Results
Scalaris sets a new bar for performance, particularly when considering the "bang for your buck" in terms of tokens:
- State-of-the-Art Accuracy: Achieved 92.2% on LongMemEval-S and 90.77% on LoCoMo. Notably, it achieved these results using the relatively small Qwen3-32B model, outperforming competitors using much larger backbones.
- Extreme Efficiency: Scalaris used only ~4,100 tokens per query compared to Hindsight’s ~43,000 tokens—a 10x reduction in context overhead with comparable or better accuracy.
- Production Impact: In a blind evaluation of 1,200 real user queries, memory-enabled responses were preferred 91.9% of the time, showing a 36% improvement in overall response quality.
Limitations and Future Work
- Memory Types: Currently, Scalaris is limited to declarative and episodic memory (facts and events). It does not yet handle procedural memory (learning new skills or workflows).
- Modality: The system is designed for text. The authors note that voice-driven environments—which require even lower latency and continuous streaming—are a target for future research.
- Implicit Negation: A known failure mode is when a user implicitly changes a preference (e.g., "I'm looking at SUVs now" after previously discussing sedans) without explicitly contradicting the old fact. Improving the system's ability to handle these "soft" updates is a priority.
Summary updated