TL;DR
This paper explores whether giving users control over their personalized news profiles (by allowing them to view and edit keywords) improves their experience and the system's performance. While users generally prefer control and transparency, the study found that actively editing profiles, especially removing keywords, significantly harmed the system's recommendation accuracy, suggesting a need for caution when implementing such features.
What Problem Does This Paper Address?
Adaptive news systems aim to deliver personalized content by understanding user interests. However, as these systems become more sophisticated, their underlying mechanisms can become opaque to users, leading to a lack of trust and control. This "black box" problem can frustrate users, making them less likely to rely on the system's recommendations. The authors propose "open user models" – where users can see and edit their profiles – as a way to increase transparency and user agency, thereby fostering trust and potentially improving the user experience. The core question is whether this increased control is ultimately beneficial or detrimental to both the user and the system's performance.
Methods
The researchers developed "YourNews," an adaptive news system that allows users to view and edit their interest profiles. The system uses a standard vector space model for user profiling and recommendation, but its novelty lies in the "open user model" interface. This interface displays keywords that constitute a user's profile, with font size indicating keyword importance. Users can add new keywords or remove existing ones.
To evaluate the impact of this feature, they conducted an experimental study with 10 graduate students. Participants were tasked with acting as information analysts, collecting relevant news articles for specific topics. They used either a "baseline" version of YourNews (standard adaptive system) or the "experimental" version (with the open user model). The study involved two sessions, simulating different points in time to assess performance over a period.
System performance was measured using precision (at ranks 10 and 20) and recall. User performance was evaluated by the quality of their collected reports, also measured by precision and recall against a manually created "ground truth." User satisfaction and perceptions of control were assessed through post-questionnaires and exit interviews.
Datasets
The study used a collection of 456 news articles gathered from 62 RSS feeds across 8 sources, covering events between September 25th and October 15th, 2006. This collection was manually annotated by three annotators to establish a "ground truth" for evaluating system and user performance. The paper does not introduce a new benchmark dataset but rather uses a curated set of real-world news articles for the experiment.
Key Insights and Ideas
- Transparency vs. Performance Trade-off: The most striking finding is the direct conflict between user preference for control and the system's objective performance. While users liked having control, it actively hurt the system's ability to recommend relevant news.
- The Harm of Keyword Removal: The study suggests that removing keywords from a profile is significantly more detrimental to recommendation accuracy than adding them. This implies users may not fully grasp the impact of removing terms, potentially deleting important signals.
- User Models are Complex: The experiment highlights that users may not have an intuitive understanding of what constitutes a "good" or "bad" keyword for a news recommendation system, leading to actions that degrade performance.
- Novelty is Crucial: The paper points out that a lack of novelty in recommendations was a significant issue. Users might try to manipulate profiles to avoid seeing repeated content, inadvertently harming the system.
- Trust Despite Poor Performance: Interestingly, users reported a relatively high level of trust in the experimental system, even when its performance was demonstrably lower. This suggests that the perception of control can influence trust, even if it doesn't lead to better outcomes.
Key Results
- System Performance: The baseline system (without open user models) consistently outperformed the experimental system in terms of precision and recall. For the National topic, the baseline achieved a precision of 0.775, while the experimental system was lower. The Health topic showed even greater performance disparity.
- User Performance: User reports generated using the baseline system also showed higher precision and recall compared to those from the experimental system.
- Keyword Manipulation Impact: A negative correlation was found between the frequency of keyword manipulation and system performance. Users who made more changes (especially removals) saw their system's precision drop significantly. For instance, "user 5" who made 16 keyword changes had a precision of only 0.304.
- User Satisfaction: Despite the performance drop, users generally rated the experimental system more positively in terms of satisfaction and control, though these differences were not always statistically significant. They particularly valued seeing the keywords behind news items.
Limitations and Future Work
The authors acknowledge that the study's findings are somewhat controversial, echoing previous work (Waern, 2004) that also found editable profiles could harm performance. They suggest that their attempt to mitigate this with immediate feedback might not have been sufficient.
Future work includes:
- Improving Novelty Control: The paper identifies a critical need to better promote novel items in recommendations, as this was a significant problem and may have driven negative user actions.
- Further Studies on Improved Systems: They plan to continue researching novelty control and run further studies on improved systems to see if the ability to change user models can be beneficial under different conditions.
- Understanding User Behavior: More research is needed to understand why users make certain profile changes and how to guide them towards actions that genuinely improve recommendations.
Summary updated