Abstract
UserAdapter is a lightweight framework designed to personalize large Transformer models for individual users using only a handful of examples. By freezing the massive pre-trained model and training only a tiny, user-specific "prefix" vector, the researchers achieved higher accuracy in sentiment analysis than full fine-tuning while adding less than 0.5% parameter overhead per user.
Introduction
In sentiment analysis, "one size fits all" models often fail because people express opinions differently—one person’s "3-star" review might be another person’s "5-star" experience. To capture these nuances, personalized models are essential. However, personalizing modern Large Language Models (LLMs) like RoBERTa presents a massive engineering hurdle. The standard approach—fine-tuning the entire model for every user—is impossible to scale; storing 125 million parameters for every single customer on a platform like Yelp or IMDB is computationally prohibitive. Alternative methods like "in-context learning" are limited by the Transformer's narrow context window. UserAdapter addresses this by providing every user a "bespoke" model that is both highly accurate and computationally cheap to store.
Methodology
The authors introduce UserAdapter, a method inspired by recent "prefix-tuning" and "adapter" research. Instead of changing the weights of the pre-trained Transformer, they "clamp" (freeze) them.
The core innovation involves:
- The User Vector: For every user, the model maintains a unique, continuous vector ($u_\theta$). This vector is prepended to the input text embeddings, acting like a virtual "prefix" token that tells the Transformer who is speaking.
- Parametrization Strategy: To make optimization more stable, they don't train the large user vector directly. Instead, they use a smaller bottleneck vector (dimension $k$) and pass it through a Multi-Layer Perceptron (MLP). This ensures the model learns a high-quality representation even with very few data points.
- Two-Stage Learning:
- Stage 1: Train a general "user-aware" model on a large dataset where many users have many reviews.
- Stage 2 (Few-Shot): When a brand-new user arrives with only a few reviews, the Transformer and classifier are frozen. Only the tiny user-specific MLP/vector is trained.
Experimental Setup
The researchers curated two specific datasets to test few-shot personalization:
- IMDB: Movie reviews with ratings from 1–10.
- Yelp: Restaurant reviews with ratings from 1–5.
Crucially, they split these into Part A (massive data for general training) and Part B (few-shot data). Part B contains users who were never seen during the initial training phase and who have written no more than 50 reviews, simulating a real-world scenario where a system must quickly adapt to a new customer.
Results and Discussion
The paper demonstrates that you don't need to rewire a model's entire "brain" to understand a specific user. A small, learned "prefix" is enough to steer the model’s internal representations toward a specific user's style and preference.
Key findings include:
- Superior Accuracy: On the IMDB dataset, UserAdapter achieved 46.15% accuracy, significantly outperforming both the base RoBERTa model (38.82%) and even a version of RoBERTa that underwent full, "heavy" fine-tuning (44.89%).
- Extreme Efficiency: UserAdapter adds less than 0.5% of the total parameters per user. This makes it feasible to store personalized "profiles" for thousands of users without exploding storage costs.
- Scaling with $k$: As shown in the empirical results, increasing the dimension of the parametrization vector ($k$) leads to steady performance improvements, allowing developers to tune the trade-off between storage and accuracy.
- Stability through Bottlenecks: Directly optimizing a large prefix vector with only 10 examples usually leads to overfitting. Using the MLP parametrization (the $k$ dimension) acts as a regularizer that makes few-shot learning viable.
Conclusion
UserAdapter bridges the gap between generalization and adaptation, achieving performance on new users comparable to those the model was originally trained on. While successful in sentiment analysis, future work could explore how this "prefix" approach scales to more complex personalized tasks like creative writing or dialogue. Additionally, combining retrieval-based methods with adaptation could further mitigate the "cold start" problem for new users.
Summary updated