Abstract
Tailoring AI responses to a user's expertise level—known as passive personalization—is often considered a primary goal for improving user experience. However, this study reveals a critical trade-off: while expertise-based adaptation can boost performance in knowledge-intensive tasks, it can also increase mental load and frustration under time pressure. Our findings suggest that passive automation is insufficient without providing users with active agency over response parameters.
Introduction
As AI assistants become ubiquitous in enterprise environments, a "one-size-fits-all" response style often fails to meet diverse user needs. A novice typically requires conceptual bridges and step-by-step instructions, whereas an expert seeks succinct, jargon-heavy answers to facilitate rapid decision-making. While prior research has explored personalization based on personality or conversation history, the impact of user expertise on task performance—especially in knowledge-intensive, timed environments—remains understudied. This paper investigates whether an AI that automatically detects and adapts to a user’s knowledge level actually aids success or if the added complexity creates new cognitive hurdles.
Methodology
The researchers developed a personalized version of an enterprise AI assistant using an agentic design. The system architecture, shown in Figure 1, comprises two primary components:
- Expertise Classification: The system estimates user expertise (Novice, Intermediate, or Expert) using four dimensions: LLM-based judgment of user queries, words-per-sentence ratios, jargon usage (benchmarked against an official product dictionary), and Flesch readability scores.
- Response Adaptation:
- Content Control: Based on pedagogical frameworks, the AI adjusts information depth. Novices receive basic explanations and tables, while experts receive direct technical descriptions.
- Stylistic Control: Utilizing DSPy’s MIPROv2 optimizer, the authors crafted "developer prompts" to ensure the AI adheres to stylistic constraints like message length and jargon density.
The study employed a within-subject design with 13 participants completing timed certification-style exams for a specific software platform. Participants were aided by both a baseline and a personalized version of the assistant. The evaluation utilized an internal query log of 1,785 entries for prompt optimization and 24 manually curated certification questions balanced for difficulty.
Results and Discussion
The experimental results highlight a "Helpfulness Paradox." While personalization improved average exam scores for novices (49.2% to 56.0%) and experts (68.3% to 74.2%), it introduced significant cognitive costs.
As illustrated in Figure 2, the personalized system successfully reduced physical and temporal demand. However, it simultaneously increased mental demand (5.00 to 5.31) and frustration (3.69 to 3.84) across the participant pool. For novices, the "helpfulness" of personalized responses often manifested as verbosity; reading longer, more detailed responses under time pressure increased frustration.
Interestingly, personalization fostered deeper engagement. Users were significantly more likely to "extrapolate" and reason through problems (34.6%) compared to the baseline (18.2%), suggesting that expertise-based adaptation helps build better mental models even if it feels more taxing. However, intermediate users did not see the same score improvements as novices or experts, suggesting that the "middle-ground" strategy for intermediates may be less effective than targeting the extremes.
Conclusion
This study demonstrates that automatic (passive) personalization cannot fully account for the context of a task. While detailed responses are beneficial in relaxed settings, they can become a liability under time pressure. These findings point toward a need for "active" personalization, where users are granted the agency to toggle response length or detail levels on the fly. Future work will explore these active steering mechanisms and develop more formal taxonomies for jargon difficulty to refine simplification for novices without sacrificing technical accuracy.
Summary updated