Personalizing LLM Agent Memory Using Biometrics

Yanhong Qian, Qingguo Meng, Shihao Ding, Xingbo Dong, Zhe Jin, Hanrui Wang, Isao EchizenarXiv 2026 · arXiv

TL;DR

Bio-Memory is a novel, biometric-aware memory architecture designed to secure and personalize LLM agents in shared multi-user environments. By augmenting structured memory notes with biometric embeddings (like faces or palmprints), it filters and restricts memory retrieval based on physical identity before performing semantic ranking. This ensures that agents provide tailored, private assistance only to the authorized person physically making the request.

What Problem Does This Paper Address?

As large language model (LLM) agents transition from single-user personal devices to shared environments—such as smart home hubs, public kiosks, or enterprise terminals—current personalization paradigms face a critical security flaw. Existing systems rely on logical identifiers like user accounts, passwords, or session states to distinguish users. However, in a shared space, an active session can easily be left open or shared.

If two different people ask a shared agent, "What is my next appointment?", a traditional semantic-search memory system cannot distinguish between them and might leak private data. This paper addresses the challenge of grounding memory retrieval in the physical identity of the current requester, transforming personalization from a simple relevance problem into a user-alignment and access-control problem.

Methods

The authors introduce Bio-Memory, which builds on top of an existing structured memory framework called A-Mem. In A-Mem, long-term memory is organized into "atomic notes" containing timestamps, keywords, context, tags, and semantic embeddings. Bio-Memory introduces a critical modification: it appends a Bio-Embedding slot to every single memory note.

Comparison of memory note structures between A-Mem and Bio-Memory

The operational pipeline consists of two main phases:

  1. Memory Construction: When an agent interacts with a user, it generates a structured note. A pre-trained biometric encoder extracts a biometric template (e.g., a face or palmprint vector) from the user, which is stored alongside the note's semantic attributes.
  2. Biometric-Aware Retrieval: At query time, the user provides a text query along with a real-time probe biometric embedding. Crucially, Bio-Memory performs biometric matching first. It filters the entire memory store using cosine similarity against a predefined verification threshold, creating a "matched candidate pool." Only then does the system sort the notes chronologically and rank them by semantic relevance to select the top-$K$ results for the LLM. If no biometric match is found, the agent falls back to its general parametric knowledge, completely withholding user-specific data.
Figure 2: Overall pipeline of Bio-Memory. (a) Note Construction: Raw agent-environment interactions are processed into structured memory notes with semantic attributes. (b) Note Storage: Each note is stored with full attributes including temporal information, textual content, contextual details, keywords, tags, semantic embeddings and biometric embeddings to support dual semantic and biometric memory management. (c) Link Generation and Memory Evolution: The LLM builds semantic links between memories and dynamically updates attributes of historical memories via new memory inputs, enabling continuous knowledge evolution and interconnected memory boxes. (d) Memory Retrieval: Biometric matching is performed prior to semantic ranking to filter valid memory candidates. Qualified memories are fed into the LLM for personalized response generation, while the model directly leverages parametric knowledge if no valid memories are available.

Datasets

To evaluate the system, the authors simulated a 10-user shared-agent environment using LoCoMo, a benchmark for long-term conversational memory. Because LoCoMo does not inherently contain biometric data, they paired conversational samples with various public face and palmprint datasets:

  • Face Benchmarks: AgeDB-30, CALFW, CFP-FF, CFP-FP, CPLFW, LFW, and VGG2-FP (using ArcFace as the encoder).
  • Palmprint Protocols: CasiaM (at 460nm, 700nm, and 850nm), IITD, MS (Blue, Green, NIR, Red), PolyU, and Tongji (using CCNet as the encoder).

Key Insights and Ideas

  • Filter First, Rank Later: A key insight is that biometric gating must happen before semantic ranking. If semantic similarity is calculated first, private notes from unauthorized users could dominate the top-$K$ retrieval slots, creating severe privacy risks.
  • Temporal Vulnerability: The authors found that "Temporal" questions (e.g., tracking a sequence of personal events) are the most sensitive to retrieval quality. When an unauthorized user (impostor) queries the system, their evaluation scores completely collapse, proving that the biometric filter successfully breaks the entire evidence chain.
  • Modality Agnostic: The architecture works effectively across entirely different physical traits (faces vs. palmprints) and varying sensor conditions, demonstrating that biometric embeddings serve as a robust, universal control signal for vector databases.

Key Results

The system was evaluated under two roles: Owner (authorized user) and Non-owner (impostor). Across all datasets, Bio-Memory successfully maintained high performance for owners while completely cutting off non-owners:

  • Face Personalization: On the CALFW dataset, the performance gap between owners and non-owners reached a massive 27.29% in F1 score and 21.15% in BLEU-1.
  • Palmprint Personalization: On the MS_Blue dataset, the corresponding gap was 25.75% in F1 and 19.22% in BLEU-1.
  • Authorized Utility: The filtering mechanism did not penalize legitimate users; owner averages remained stable and strong across all benchmarks, showing that the biometric threshold successfully preserves useful memories.

Limitations and Future Work

While Bio-Memory demonstrates strong performance, the paper notes or implies a few limitations:

  • Simulated Evaluation: The experiments rely on synthetically binding text datasets (LoCoMo) to biometric datasets rather than testing on a live, real-world deployment with multi-modal hardware.
  • Overhead and Latency: The paper does not deeply explore the computational overhead or latency introduced by performing continuous online biometric matching across massive memory graphs.
  • Privacy of Biometrics: Storing biometric embeddings directly inside an agent's memory graph raises secondary privacy concerns regarding how those biometric templates themselves are secured against adversarial extraction.

Summary updated

Abstract

Personalized memory helps LLM agents deliver stable, tailored assistance by storing and reusing user-specific data across interactions. In multi-user scenarios, however, retrieval must consider not only semantic similarity but also whether the current requester matches the identity associated with the stored memory. We propose Bio-Memory, a biometric-aware memory architecture that conditions memory retrieval on both semantic similarity and biometric matching. Built on top of A-Mem, Bio-Memory augments each atomic memory note with a biometric embedding and uses biometric matching to form the retrieval candidate pool before semantic ranking. We evaluate Bio-Memory on LoCoMo in a 10-user shared-agent setting over 7 face benchmarks and 10 palmprint protocols. Across datasets, Bio-Memory consistently separates owner and non-owner queries. Under face-based personalization, the largest average gap reaches 27.29% / 21.15% in F1 / BLEU-1 on CALFW; under palmprint-based personalization, the corresponding gap is 25.75% / 19.22% on MS_Blue. These results support biometrics as a practical control signal for personalized memory retrieval in shared environments.
BibTeX
@misc{qian2026personalizing,
  author = {Yanhong Qian and Qingguo Meng and Shihao Ding and Xingbo Dong and Zhe Jin and Hanrui Wang and Isao Echizen},
  title = {{Personalizing LLM Agent Memory Using Biometrics}},
  year = {2026},
  eprint = {2609.08558},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.08558},
}
Figure