To truly understand a user—whether predicting their personality, tracking their mental health, or recommending their next purchase—we need to look at their history. Human behavior unfolds over months and years, generating extensive sequences of interactions, posts, and transactions. However, when these long-term longitudinal histories meet modern Transformer-based architectures, they hit a fundamental barrier: the context window bottleneck.
As outlined by Likang Wu et al. in A survey on large language models for recommendation, the finite context window of language models forces a strict trade-off between the depth of historical context and the computational cost of processing long sequences. Because standard attention mechanisms scale quadratically with sequence length, models cannot simply ingest a user's entire lifetime of data.
The Destructive Nature of Truncation
The simplest workaround to context limitations is truncation—discarding older data to fit a fixed token limit. However, human behavior contains dependencies that span far beyond a short temporal window. Slicing the history causes models to miss critical longitudinal patterns.
For example, in recommendation systems, Kyuyong Shin demonstrates in Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning that truncation discards historical preferences that fall outside the maximum token limit, losing vital signals from a user's distant past. Furthermore, Nian Li et al. show in Modeling User Fatigue for Sequential Recommendation that truncated interaction histories prevent models from identifying long-term interest decay (user fatigue), making it impossible to distinguish between a temporary shift in attention and a permanent change in preference.
In human-centered NLP tasks like personality and psychological profiling, the loss is equally severe. Niranjan Balasubramanian et al. note in Hierarchical Modeling for User Personality Prediction: The Role of Message-Level Attention that dividing data into smaller chunks fails to capture full longitudinal context over long durations. This is echoed by Matej Gjurkovic et al. in PANDORA Talks: Personality and Demographics on Reddit, who show that effective personality detection requires analyzing long-term linguistic histories rather than short text snapshots.
To solve this, a new research direction has emerged focused on bypassing context windows through architectural changes, memory systems, and compression.
Mechanism 1: Hierarchical Aggregation and Patching
Instead of feeding raw, token-by-token interaction logs into a flat Transformer, researchers use hierarchical structures to compress the effective sequence length. The intuition is simple: individual words or clicks are noisy, but structured groups (like messages or sessions) carry dense semantic signals.
Raw Tokens: [w1, w2, w3] [w4, w5, w6] [w7, w8, w9]
\ / /
Item Patches: [ Patch 1 ] [ Patch 2 ] /
\ / /
Session Patches: [ Session Patch 1 ] <-- Massive Context Reduction
Niranjan Balasubramanian et al. pioneered this with a two-level attention model (word-level and message-level) that learns to prioritize high-signal messages over low-signal noise. For large language models, Jiayi Liao et al. introduced Multi-Grained Patch Training for Efficient LLM-based Recommendation, which aggregates textual tokens into "item patches" and then "session patches." This hierarchical compression achieves up to a 13.6x reduction in token usage, allowing the model to preserve long-term preferences that would otherwise be truncated. Similarly, Matthew Matero et al. developed MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection to explicitly model sequences of user messages rather than isolated words, capturing broader social context efficiently.
Mechanism 2: Cognitive Memory and Adaptive Retrieval
Another approach mimics human memory architectures, separating immediate processing from long-term storage. Instead of forcing the model to read the entire history during every inference step, relevant historical context is retrieved dynamically.
The authors of Cognitive Personalized Search Integrating Large Language Models with an Efficient Memory Mechanism (Yujia Zhou et al.) implement this by organizing user history into sensory, working, and long-term memory layers. This mitigates the computational cost of analyzing entire interaction logs for every query.
To optimize the speed of this retrieval, Yingyi Zhang et al. propose Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval. This framework mimics the brain's dual-process memory, adaptively switching between fast, one-shot "familiarity" lookups and deep, iterative "recollection" chains to avoid the massive latency of processing a user's entire history.
User Query ---> [ Familiarity Lookup ] -------------> Fast Response
\ (If complex)
---> [ Recollection Chain ] ---> Deep Personalized Response
^
[ External Long-Term Memory ]
Crucially, standard semantic retrieval (like BM25 or vector similarity) does not always select the history that helps the model make a decision. To fix this, Yang Zhang et al. developed Leveraging Memory Retrieval to Enhance LLM-based Generative Recommendation, which trains a specialized retriever to select past interactions based explicitly on how much they improve the LLM's downstream prediction accuracy.
Mechanism 3: Dense User Embeddings as Soft Prompts
If text histories are too bulky, can we compress a user's entire behavioral trajectory into a single dense vector?
Langming Liu et al. systematically evaluated this paradigm in UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answering. They found that compressing raw text histories into dense user embeddings—which act as "soft prompts"—yields an 8x to 19x reduction in token usage while matching the performance of raw text prompts.
Similarly, Nikita Soni 0002 et al. explore this in Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks, demonstrating that tracking user history explicitly via Human Language Models (HuLMs) produces superior hidden-state embeddings that capture stable, long-term psychological traits better than simple token averaging.
Mechanism 4: Decoupling and Prefix Caching
In real-world production systems, models must operate under strict millisecond-level latency budgets. Even if a model can accept a long context window, processing that context at inference time is often too slow.
Traditional: [ Process Long User History + Item Context ] ---> Compute Attention (Slow)
Decoupled: [ Pre-compute History Prefix ] --(Cached NPU Memory)--> [ Rank Items ] (Fast)
To solve this operational bottleneck, Jiarui Wang et al. developed RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference. Their system decouples user-history processing from item ranking by pre-calculating and caching user-behavior "prefixes" in NPU memory during earlier pipeline stages. This engineering optimization allows production models to utilize 1.5× longer sequences and boosts throughput by 3.6× without violating real-time latency constraints.
Key Foundational Work
Hierarchical Modeling for User Personality Prediction: The Role of Message-Level Attention (2020) — contributed 2 concept elements to this direction.
SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions (2018) — contributed 1 concept elements to this direction.