Abstract
Information retrieval (IR) in dynamic data streams is a crucial task, as shifts in data distribution degrade the performance of AI-powered IR systems. To mitigate this issue, memory-based continual learning has been widely adopted for IR. However, existing methods rely on a fixed set of queries with ground-truth documents, which limits generalization to unseen data, making them impractical for real-world applications. To enable more effective learning with unseen topics of a new corpus without ground-truth labels, we propose CREAM, a self-supervised framework for memory-based continual retrieval. CREAM captures the evolving semantics of streaming queries and documents into dynamically structured soft memory and leverages it to adapt to both seen and unseen topics in an unsupervised setting. We realize this through three key techniques: fine-grained similarity estimation, regularized cluster prototyping, and stratified coreset sampling. Experiments on two benchmark datasets demonstrate that CREAM exhibits superior adaptability and retrieval accuracy, outperforming the strongest method in a label-free setting by 27.79% in Success@5 and 44.5% in Recall@10 on average, and achieving performance comparable to or even exceeding that of supervised methods.
Introduction
Most modern search engines and AI-powered retrieval systems, such as those used in Retrieval-Augmented Generation (RAG), are typically trained on static snapshots of data. However, real-world information exists in a "dynamic stream" where topics evolve and the semantic meaning of terms shifts based on context. When new data arrives, retraining models from scratch is computationally prohibitive, while incremental updates often lead to catastrophic forgetting—the loss of previously acquired knowledge.
Existing solutions generally rely on "hard memory," which stores a fixed set of human-labeled query-document pairs. This creates a significant bottleneck: in fast-moving web data streams, human labeling cannot keep pace with the emergence of new topics. To overcome this, we introduce CREAM (Continual REtrieval with Adaptive Soft Memory), a framework that replaces static hard memory with a dynamic, self-supervised soft memory structure.
Methodology
CREAM enables models to learn on the fly without human intervention through three primary technical pillars:
- Fine-grained Similarity Estimation: Rather than compressing a document into a single vector, CREAM utilizes token-level interactions. This allows the system to detect subtle semantic shifts in how specific words are used as new topics emerge.
- Regularized Cluster Prototyping: To maintain efficiency, CREAM groups similar documents and queries into clusters. Using Random Projection Locality Sensitive Hashing (RP-LSH), the system creates fixed-size "prototypes" (centroids) for these clusters, reducing computational overhead by over 1,000x compared to raw token comparisons.
- Stratified Coreset Sampling: To facilitate training without labels, CREAM selects a diverse "coreset" of representative queries and documents from its soft memory. It then employs self-supervised contrastive learning, teaching the model to pull queries closer to their most similar cluster documents while pushing them away from unrelated clusters.
Experimental Evaluation
We evaluated CREAM on two major benchmarks, LoTTE and MSMARCO, modified to simulate a 10-session streaming environment where topics appear, overlap, and disappear over time.
Key Results
- Superior Accuracy: CREAM significantly raised the bar for label-free continual retrieval, outperforming the strongest unsupervised baseline by 27.79% in Success@5 and 44.5% in Recall@10 on average.
- Rivaling Supervised Models: Surprisingly, CREAM often performed better than supervised methods. While supervised models are anchored to a fixed set of "old" queries, CREAM’s soft memory is topic-aware, adapting its internal structure to the current data distribution.
- Robustness: Ablation studies confirmed that fine-grained similarity was the most critical component; its removal resulted in a performance drop of nearly 45%.
Conclusion and Future Work
CREAM demonstrates that self-supervised "soft" memory structures are more effective for dynamic retrieval than traditional "hard" labeled memory. By storing the relationships between topics rather than specific pairs, the system can gracefully decay outdated information while making room for emerging trends.
While highly effective, the authors noted that query selection currently accounts for nearly 50% of processing time. Future research will focus on scaling CREAM to broader tasks like real-time summarization and developing hierarchical memory structures to handle more complex, nested topic shifts.
Summary updated