TL;DR
Researchers have found that personalizing the order of sentences in a recommendation explanation—based on how a specific user has interpreted information in the past—significantly boosts their objective comprehension. In a study of 122 participants, explanations tailored to a user's unique "interpretive profile" led to an 86.5% comprehension rate, outperforming population-based averages and even a target-aware Large Language Model (LLM) approach.
What Problem Does This Paper Address?
Most personalized recommendation systems focus on what to tell you (content) or how to say it (style). For example, if you like action movies, the system highlights the stunts. However, people also differ in how they mentally organize information. One person might always look for the price first, while another might prioritize location and then check if the event is family-friendly.
The authors argue that we leave a "paper trail" of these mental habits in our written interpretations. By analyzing how a user previously described various services (like meal kits or phone plans), can we predict the best way to order information in a new explanation to make it easier to digest? The core challenge is moving beyond simple preferences to capture "interpretive regularities"—the structural habits of a user’s mind.
Methods
The researchers used the Interpretive Schema Manifestation Approach (ISMA) to build historical user profiles. They represented a user's past interpretations across three dimensions:
- Information Selection: Which of the six functional roles (e.g., cost, access, core attributes) did the user mention?
- Relative Salience: How much prominence did the user give to each role?
- Relational Organization (Co-mobilization): Which pieces of information did the user tend to mention together in the same sentence?
To test this, they conducted a within-participant study. First, they collected four written interpretations from each participant to build a "Historical Profile." Then, they presented new recommendations using four different sentence-ordering procedures:
- OWN: Sentences ordered based on the user's own historical profile.
- DONOR: Ordered based on a random other user's profile.
- POP: Ordered based on the average of the entire population.
- PROFILE-LLM: An LLM (Mistral-7B) was given the user's profile and the target context to determine the "best" order.
Crucially, the content and wording remained identical across all conditions; only the sequence of the six sentences changed.
Datasets
The study did not use a pre-existing public dataset. Instead, the authors created a custom dataset through a two-phase experimental process:
- Profile Construction Phase: 122 participants provided 488 written interpretations across four "source" scenarios (outdoor activity, meal-kit, mobile plan, home cleaning).
- Evaluation Phase: The same participants were tested on four "target" scenarios (watching a film, choosing a cultural activity, a cooking workshop, and a musical performance).
Key Insights and Ideas
- Structure is a Signal: A user's past writing reveals stable patterns in how they prioritize and group information. These patterns are "actionable signals" for UI/UX design.
- Comprehension $\neq$ Acceptance: This is perhaps the most striking "aha moment." While personalizing the order made people understand the explanation much better, it didn't make them more likely to accept the recommendation. This suggests that better understanding doesn't automatically lead to more persuasion—it might even help a user realize a recommendation isn't right for them.
- Rules vs. LLMs: A simple, deterministic rule based on a user's profile (OWN) outperformed a sophisticated LLM (PROFILE-LLM) that had access to the same profile plus the specific context of the recommendation. This suggests that for structural personalization, direct mapping of user habits can be more effective than "black box" reasoning.
Key Results
The primary metric was objective comprehension, measured by factual and integration questions after the explanation was shown and then removed.
- OWN achieved 86.5% accuracy, significantly higher than all other conditions.
- PROFILE-LLM followed at 73.0%.
- DONOR (66.0%) and POP (63.9%) performed the worst, showing that generic or "mismatched" ordering actually hinders understanding.
The advantage of the OWN condition held true for both simple factual questions and more complex "integration" questions that required combining multiple pieces of information.
Limitations and Future Work
The authors acknowledge several boundaries to their findings:
- Scope of Personalization: The study only varied sentence order. It didn't look at how changing the wording or adding/removing content might interact with the ordering.
- Short-term vs. Long-term: The profiles were built from just four interpretations. It’s unclear how these interpretive habits evolve over longer periods or across very different domains.
- Cognitive Mechanism: While comprehension improved, the study doesn't pinpoint why (e.g., reduced cognitive load, better alignment with memory structures).
- Future Directions: The researchers suggest exploring "richer adaptive interfaces" where both content and organization are personalized, and testing these methods in real-world, long-term deployments rather than controlled lab settings.
Summary updated