TL;DR
Axis-LoRA is a lightweight adapter framework designed to fix the "axis-blindness" of standard embedding models in personal memory retrieval. By inserting four axis-specific LoRA branches (Temporal, Spatial, Entity, and Event) into the late layers of a model and using a per-query gate to combine them, it significantly improves the ability to distinguish memories by time, place, person, and action without sacrificing general retrieval performance.
What Problem Does This Paper Address?
As Large Language Models (LLMs) become more integrated into our daily lives, they are increasingly tasked with answering questions about our personal experiences—like "Where did I park at the Boston conference?" or "What did I discuss with Marco on Tuesday?"
To answer these, a system must retrieve the correct memory from a massive corpus. However, the authors discovered that current state-of-the-art embedding models (like BERT-based encoders or LLM-based retrievers) are surprisingly bad at this. They suffer from axis-asymmetric under-exposure: they are optimized for general topic overlap but fail to distinguish between memories that differ only in specific "axes" like time or location. For example, swapping "Tuesday" for "Friday" in a memory barely shifts its embedding vector, making it nearly impossible for a retriever to find the specific "Tuesday" meeting among many similar ones.
Methods
The authors first performed a diagnostic "probe" to find where the signal for these axes goes. They found that while the final output of the model is "blind" to these changes, the signal actually exists deep inside the model, concentrated in specific attention heads within the last 20% of the transformer layers.
Based on this, they developed Axis-LoRA:
- Per-Axis Adapters: They added four independent Low-Rank Adaptation (LoRA) branches to the late layers of the frozen base model. Each branch is trained to specialize in one axis: Temporal, Spatial, Entity, or Event.
- Query-Conditioned Gating: Since a query usually only targets one or two axes (e.g., "Where" targets Spatial), they added a small MLP gate. This gate uses a
sparsemaxfunction to output weights that amplify relevant axes and set irrelevant ones to exactly zero, preventing noise. - Hybrid Training: They trained the model using a contrastive loss on a synthetic counterfactual dataset while using a knowledge-distillation (KD) anchor to a general retrieval dataset (MS MARCO). This ensures the model learns personal memory nuances without "forgetting" how to do general search.
Datasets
- AxisCF (New): A synthetic corpus of "axis-isolated counterfactuals." For every memory, they generated versions where only one axis value was changed (e.g., changing the person's name but keeping everything else identical). This forces the model to learn the difference.
- LongMemEval: A benchmark for long-term multi-session chat histories.
- ATM-Bench: A multimodal personal memory benchmark covering images, videos, and emails.
- LoCoMo: A benchmark for very long-term conversations spanning multiple months.
Key Insights and Ideas
- The "Temporal" Gap: Models are uniquely bad at time. Temporal information is often at the "noise floor" of embeddings—swapping a date is seen as less significant by the model than swapping a random vocabulary word.
- Signal is Latent, not Lost: The information isn't erased during encoding; it's just "compressed" in a way that standard cosine similarity can't see. The signal lives in the attention patterns, which is why internal adapters (LoRA) work better than external projection heads.
- Late Layer Concentration: Axis-sensitive computation isn't spread throughout the model; it's a late-stage refinement. This allows for very efficient adaptation by only touching the final few layers.
Key Results
Axis-LoRA was tested across three different base models (MiniLM-L6-v2, bge-base-en-v1.5, and Qwen3-Embedding-0.6B).
- Consistent Gains: It improved the 4-benchmark mean Recall@10 (R@10) by +1.28 to +1.56 percentage points.
- Hard Queries: The biggest improvements were seen on the "ATM-Bench Hard" split, which features complex, multi-evidence queries, with gains up to +3.8 pp.
- Efficiency: These results were achieved by training only a tiny fraction of parameters (<0.6% of the base model) on a small dataset of just ~4,200 triplets.
- Generalization: The model maintained its general-purpose retrieval strength, showing only a negligible drop (~0.5 pp) on the standard MTEB benchmark.
Limitations and Future Work
- Fixed Axis Schema: The current four axes are tied to existing benchmarks. Other domains (like medical or legal) might require different axes.
- Explicit vs. Implicit Cues: The training currently focuses on explicit values (e.g., "2024-06-15"). It may be less effective for implicit cues like "last weekend" or "the meeting before lunch."
- Static Gating: The authors suggest that moving from a simple MLP gate to more complex Mixture-of-Experts (MoE) routing could further improve performance.
- Per-User Personalization: A natural next step is to see if this "recipe" can be used to quickly adapt a model to a single user's specific interaction style.
Summary updated