Introduction: The Challenge of Human-Centered Text Understanding
Human-Centered Text Understanding (HCTU) recognizes that the meaning of a text often depends on its author. In personalized sentiment analysis, for example, a "3-star" review from a harsh critic conveys a different sentiment than the same rating from a lenient user. Adapting Large Language Models (LLMs) for such personalization faces three primary hurdles:
- Scalability: Storing unique weights or even standard LoRA adapters for millions of users is computationally prohibitive.
- Cold-Start: Models often struggle with new or anonymous users (zero-shot/few-shot scenarios) because they lack historical preference data.
- Efficiency: Traditional knowledge injection methods typically require full-model fine-tuning, which is too expensive for large-scale Pre-trained Language Models (PLMs).
Methodology: PLoRA and Plug-and-Play
The authors introduce PLoRA (Personalized Low-Rank Adaptation), a framework that bridges the gap between Personalized Knowledge Injection (PKI) and Parameter-Efficient Fine-Tuning (PEFT).
Shared Low-Rank Structure
Instead of separate adapters, PLoRA utilizes a shared architecture where user attributes are converted into a unified embedding ($p$). This embedding is combined with the text representation ($h$) and passed through a shared output projection matrix. This coupling allows the model to learn a cohesive relationship between text and persona while drastically reducing parameter overhead.
Plug-and-Play (PnP) for Cold-Starts
For new users, PLoRA employs a "Plug-and-Play" approach: the model weights remain frozen, and only the user embedding ($p$) is updated via a quick backpropagation step. To ensure the model remains robust for anonymous users, the authors incorporate:
- PDropout (Personalized Dropout): Randomly masking user tokens during training to maintain general text-understanding capabilities.
- MIM (Mutual Information Maximization): A strategy to align task-specific and human-centered representations, acting as a regularizer to improve generalization.
Key Insights
- Weight Sharing as Integration: By sharing the output matrix between the task-specific LoRA and the user-specific injection, personalization becomes an integral part of the task adaptation rather than an isolated feature.
- The User as an "Access Token": In the PnP framework, the user embedding acts like a dynamic access token, allowing for seamless deployment where only the token is swapped per user without altering the underlying model.
- Decoupling for Generalization: Strategies like MIM and PDropout effectively decouple personalization from general understanding, allowing the model to be highly personalized for known users while remaining accurate for strangers.
Experimental Results
PLoRA was evaluated across benchmark datasets including IMDB, YELP, GDRD, and PPR. It consistently outperformed state-of-the-art methods like MA-BERT and UserAdapter while requiring 30x to 50x fewer trainable parameters than full-model fine-tuning.
As shown in the results, PLoRA demonstrates robust adaptation in few-shot scenarios, reaching competitive performance with as few as 15 samples. In zero-shot (anonymous) settings, it maintains high accuracy where other personalized models typically degrade.
Limitations and Future Work
While highly effective for sentiment analysis, the authors note that HCTU encompasses many other tasks, such as personalized dialogue or recommendation, which remain to be explored. Future work could also extend PLoRA beyond multi-head attention layers to other components like LayerNorm or hybrid structures, and investigate its utility in broader text-generative tasks.
Summary updated