Scalaris: A Scalable Enterprise Memory Architecture for Conversational AI Assistants

Devarsh Gandhi, Sarthak Pawar, Hemprasad Badgujar, Vaibhav Rathore, Siddharth Sureka, Neha BhargavaarXiv 2026 · arXiv

TL;DR

Scalaris is a novel memory architecture for enterprise AI assistants that breaks the traditional trade-off between retrieval accuracy and token efficiency. By treating user interactions (relational/long-term) and assistant responses (contextual/short-term) as fundamentally different data types, it achieves state-of-the-art accuracy while using 10x fewer retrieved tokens than leading systems like Hindsight.

What Problem Does This Paper Address?

Building a conversational AI that "remembers" you across months of interactions is hard. At enterprise scale, you face a brutal accuracy-efficiency trade-off:

  1. High Accuracy, High Cost: Systems like Hindsight retrieve massive chunks of raw history to ensure the model has all the facts. This leads to "context bloat," driving up inference costs and latency.
  2. Low Cost, Low Accuracy: Systems like Zep use aggressive summarization or lightweight graphs, but often miss the specific nuances needed to answer complex, multi-session questions.

Furthermore, most existing systems treat every message the same. The authors argue this is a mistake: user messages contain evolving facts about preferences, while assistant messages are often verbose, contextual, and become "stale" quickly (especially in fast-moving fields like finance).

Methods

Scalaris introduces a source-aware memory organization that splits the ingestion and storage pipeline based on who is speaking:

  • User Interactions (Long-term): These are processed into a Temporal Knowledge Graph (built on an enhanced version of Graphiti). It extracts atomic facts and entities, tracking their validity over time (e.g., "John prefers growth stocks" might change to "John prefers value stocks").
  • Assistant Responses (Short-term): Instead of storing full responses, Scalaris breaks them into atomic memory units. These are filtered for "low-value" content (like greetings) and stored in a Vector Store. This allows the system to retrieve only the specific informative snippet of a past answer rather than the whole verbose transcript.
  • The Retrieval Pipeline: When a new query comes in, Scalaris first performs Query Reformulation to resolve pronouns (e.g., changing "What about that one?" to "What about the Toyota Corolla?"). It then executes a hybrid search across the graph and vector stores using semantic, lexical, and graph-traversal techniques.
Figure 1: Overview of Scalaris’s source-aware memory organization. User interactions are processed to main- tain relevant facts, relevant entities, and user and session summaries in a temporal knowledge graph. Assistant re- sponses are decomposed into independently retrievable atomic units and stored in a vector store. In parallel, both user interactions and assistant responses are pre- served verbatim in a shared history store.

Datasets

The authors evaluated Scalaris on two primary long-term memory benchmarks and one real-world production set:

  • LongMemEval-S: 500 questions testing long-term recall and reasoning.
  • LoCoMo: 1,986 questions across multi-session conversations.
  • Internal Production Data: 1,200 real-world interactions from Motilal Oswal’s proprietary financial assistants to validate performance in the wild.

Key Insights and Ideas

  • Functional Asymmetry: The core "aha moment" is that user inputs and assistant outputs serve different roles. User inputs are relational (building a profile), while assistant outputs are informational (providing data points). Tailoring the storage medium to these roles is what enables the 10x efficiency gain.
  • Atomic Assistant Memory: By fragmenting assistant responses into self-contained units, the system avoids the "Lost in the Middle" problem where LLMs struggle with overly long contexts.
  • Temporal Awareness: The use of a bi-temporal model (tracking when a fact was learned vs. when it is valid) prevents the assistant from retrieving "stale" information, which is critical for enterprise applications like finance.
Figure 2: An illustrative instance of context retrieval in Scalaris. Long-term memory supplies the user’s preference for fuel economy together with prior Honda Civic interactions, while short-term memory supplies the Relevant Assistant Messages, session summary, and recent turns (only the latest exchange is shown, for visual clarity). Together these resolve the referent, let the agent determine that only Corolla information must be retrieved, and enable personalization by focusing on fuel economy and including trade-offs.

Key Results

Scalaris sets a new bar for performance, particularly when considering the "bang for your buck" in terms of tokens:

  • State-of-the-Art Accuracy: Achieved 92.2% on LongMemEval-S and 90.77% on LoCoMo. Notably, it achieved these results using the relatively small Qwen3-32B model, outperforming competitors using much larger backbones.
  • Extreme Efficiency: Scalaris used only ~4,100 tokens per query compared to Hindsight’s ~43,000 tokens—a 10x reduction in context overhead with comparable or better accuracy.
  • Production Impact: In a blind evaluation of 1,200 real user queries, memory-enabled responses were preferred 91.9% of the time, showing a 36% improvement in overall response quality.
Comparison of tokens vs. accuracy

Limitations and Future Work

  • Memory Types: Currently, Scalaris is limited to declarative and episodic memory (facts and events). It does not yet handle procedural memory (learning new skills or workflows).
  • Modality: The system is designed for text. The authors note that voice-driven environments—which require even lower latency and continuous streaming—are a target for future research.
  • Implicit Negation: A known failure mode is when a user implicitly changes a preference (e.g., "I'm looking at SUVs now" after previously discussing sedans) without explicitly contradicting the old fact. Improving the system's ability to handle these "soft" updates is a priority.

Summary updated

Abstract

Enterprise conversational systems require memory to personalize interactions, retain information across sessions, and maintain conversational coherence. At enterprise scale, however, memory must not only retrieve relevant information accurately but also provide it efficiently to the answer-generation model. Existing memory systems face a trade-off between retrieval accuracy and context efficiency. Retrieving more context can improve recall but increases inference cost and latency. We present Scalaris, a scalable enterprise memory architecture that addresses this trade-off through a unified long-term and short-term memory design. We propose that user interactions and assistant responses possess distinct characteristics and roles in a conversation: user interactions capture evolving knowledge about the user and provide value across sessions, whereas assistant responses must be stored efficiently so that only information relevant to the current query is retrieved. We therefore treat them separately and support each with specialized representations and ingestion/retrieval pipelines. Scalaris reaches 92.2 % accuracy on LongMemEval-S and 90.77 % on LoCoMo using Qwen3-32B, state-of-the-art results on both benchmarks, while supplying the answer-generation model with approximately 10 times fewer retrieved context tokens than Hindsight, a leading memory system for conversational AI. When integrated with our proprietary closed-source AI assistants, Scalaris improved overall response quality by 36 %. Together, these results establish Scalaris as a cost-effective framework for building scalable, personalized conversational AI systems.
BibTeX
@misc{gandhi2026scalaris,
  author = {Devarsh Gandhi and Sarthak Pawar and Hemprasad Badgujar and Vaibhav Rathore and Siddharth Sureka and Neha Bhargava},
  title = {{Scalaris: A Scalable Enterprise Memory Architecture for Conversational AI Assistants}},
  year = {2026},
  eprint = {2611.02612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02612},
}
Figure