Axis-LoRA: Probing and Adapting Embedding Models for Personal-Memory Retrieval

Dohoon Kim, Junho Park, Hyunwoong Bae, Seyoon Kim, Taesup MoonarXiv 2026 · arXiv

TL;DR

Axis-LoRA is a lightweight adapter framework designed to fix the "axis-blindness" of standard embedding models in personal memory retrieval. By inserting four axis-specific LoRA branches (Temporal, Spatial, Entity, and Event) into the late layers of a model and using a per-query gate to combine them, it significantly improves the ability to distinguish memories by time, place, person, and action without sacrificing general retrieval performance.

What Problem Does This Paper Address?

As Large Language Models (LLMs) become more integrated into our daily lives, they are increasingly tasked with answering questions about our personal experiences—like "Where did I park at the Boston conference?" or "What did I discuss with Marco on Tuesday?"

To answer these, a system must retrieve the correct memory from a massive corpus. However, the authors discovered that current state-of-the-art embedding models (like BERT-based encoders or LLM-based retrievers) are surprisingly bad at this. They suffer from axis-asymmetric under-exposure: they are optimized for general topic overlap but fail to distinguish between memories that differ only in specific "axes" like time or location. For example, swapping "Tuesday" for "Friday" in a memory barely shifts its embedding vector, making it nearly impossible for a retriever to find the specific "Tuesday" meeting among many similar ones.

Figure 1: Overview. (a) Failure case in personal-memory retrieval. A frozen retriever groups same-content memories into a single embedding cluster, where shared content tokens place candidates at near-equal cosine distance. An axis cue in the query (here, the time axis) does not separate the gold from content-similar distractors that mismatch on the axis, so the gold falls below them in the ranked list. (b) Axis-LoRA. Four axis-conditional LoRA adapters (T/S/E/Ev) attach to the last 20% of layers of the frozen embedding model and produce per-axis representations ϕA(q), ϕA(m). A per-query gate (MLP →sparsemax) outputs routing weights wA with exact zeros on inactive axes. The final score is the frozen-base similarity plus a gated per-axis term.

Methods

The authors first performed a diagnostic "probe" to find where the signal for these axes goes. They found that while the final output of the model is "blind" to these changes, the signal actually exists deep inside the model, concentrated in specific attention heads within the last 20% of the transformer layers.

Based on this, they developed Axis-LoRA:

  1. Per-Axis Adapters: They added four independent Low-Rank Adaptation (LoRA) branches to the late layers of the frozen base model. Each branch is trained to specialize in one axis: Temporal, Spatial, Entity, or Event.
  2. Query-Conditioned Gating: Since a query usually only targets one or two axes (e.g., "Where" targets Spatial), they added a small MLP gate. This gate uses a sparsemax function to output weights that amplify relevant axes and set irrelevant ones to exactly zero, preventing noise.
  3. Hybrid Training: They trained the model using a contrastive loss on a synthetic counterfactual dataset while using a knowledge-distillation (KD) anchor to a general retrieval dataset (MS MARCO). This ensures the model learns personal memory nuances without "forgetting" how to do general search.

Datasets

  • AxisCF (New): A synthetic corpus of "axis-isolated counterfactuals." For every memory, they generated versions where only one axis value was changed (e.g., changing the person's name but keeping everything else identical). This forces the model to learn the difference.
  • LongMemEval: A benchmark for long-term multi-session chat histories.
  • ATM-Bench: A multimodal personal memory benchmark covering images, videos, and emails.
  • LoCoMo: A benchmark for very long-term conversations spanning multiple months.

Key Insights and Ideas

  • The "Temporal" Gap: Models are uniquely bad at time. Temporal information is often at the "noise floor" of embeddings—swapping a date is seen as less significant by the model than swapping a random vocabulary word.
  • Signal is Latent, not Lost: The information isn't erased during encoding; it's just "compressed" in a way that standard cosine similarity can't see. The signal lives in the attention patterns, which is why internal adapters (LoRA) work better than external projection heads.
  • Late Layer Concentration: Axis-sensitive computation isn't spread throughout the model; it's a late-stage refinement. This allows for very efficient adaptation by only touching the final few layers.

Key Results

Axis-LoRA was tested across three different base models (MiniLM-L6-v2, bge-base-en-v1.5, and Qwen3-Embedding-0.6B).

  • Consistent Gains: It improved the 4-benchmark mean Recall@10 (R@10) by +1.28 to +1.56 percentage points.
  • Hard Queries: The biggest improvements were seen on the "ATM-Bench Hard" split, which features complex, multi-evidence queries, with gains up to +3.8 pp.
  • Efficiency: These results were achieved by training only a tiny fraction of parameters (<0.6% of the base model) on a small dataset of just ~4,200 triplets.
  • Generalization: The model maintained its general-purpose retrieval strength, showing only a negligible drop (~0.5 pp) on the standard MTEB benchmark.

Limitations and Future Work

  • Fixed Axis Schema: The current four axes are tied to existing benchmarks. Other domains (like medical or legal) might require different axes.
  • Explicit vs. Implicit Cues: The training currently focuses on explicit values (e.g., "2024-06-15"). It may be less effective for implicit cues like "last weekend" or "the meeting before lunch."
  • Static Gating: The authors suggest that moving from a simple MLP gate to more complex Mixture-of-Experts (MoE) routing could further improve performance.
  • Per-User Personalization: A natural next step is to see if this "recipe" can be used to quickly adapt a model to a single user's specific interaction style.

Summary updated

Abstract

As LLMs are increasingly asked questions about a user's personal experiences, retrieval must distinguish memories by time, place, person, and event. Current personal-memory pipelines rely on frozen dense embedding models to embed memories for retrieval, but we find that these models do not capture the four axes well. Swapping a memory's date or place barely shifts its embedding, and a trained projection on top of the frozen embedding does not recover the missing signal. The problem lives inside the model, where each axis is encoded by a distinct attention-head pattern. Guided by these findings, we propose Axis-LoRA, a per-axis adapter that places four LoRA branches inside the embedding model and combines them via a per-query gate. Across three standard embedding models evaluated on LongMemEval, ATM-Bench, and LoCoMo, Axis-LoRA improves 4-benchmark mean R@10 by $+1.28$ to $+1.56$~pp while training only a small fraction of the base parameters on a small axis-counterfactual training set. Code is available at https://github.com/dohoonkim-ai/axis-lora.
BibTeX
@misc{kim2026axis,
  author = {Dohoon Kim and Junho Park and Hyunwoong Bae and Seyoon Kim and Taesup Moon},
  title = {{Axis-LoRA: Probing and Adapting Embedding Models for Personal-Memory Retrieval}},
  year = {2026},
  eprint = {2611.07820},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07820},
}
Figure