TL;DR
This paper introduces PERCY, a system that delivers tailored and coherent information from corporate websites, akin to a personalized brochure. An empirical study found that users preferred PERCY's output over traditional search engine results, indicating that this approach is more engaging and understandable for users seeking information.
What Problem Does This Paper Address?
Navigating corporate websites to find specific information can be challenging. Traditional methods like searching and browsing often yield overwhelming amounts of data that aren't tailored to individual user needs or presented in an easily digestible format. This makes it difficult for users to quickly grasp relevant information and for corporations to effectively communicate their offerings. The authors aim to address this by exploring whether a system that combines user modeling with coherent, tailored document delivery can provide a superior user experience compared to standard search engines.
Methods
The core of the paper is the PERCY system, which integrates information retrieval with user modeling and discourse-oriented technologies. PERCY takes a user's query and a modest user profile (including stereotypical information like job position and individual information like interests) to construct a coherent, brochure-like document. This involves:
- Virtual Document Planner (VDP): This component creates a "discourse tree" that dictates the information to be included and its organization. It uses metadata to pull relevant snippets from existing web pages.
- Discourse Planner: Modeled on existing work, this planner uses a library of "discourse rules" to ensure the information is presented logically and coherently, mimicking how information is structured in traditional brochures.
- Presentation Planner: This module decides on the amount of information to present and tailors the presentation to the chosen delivery medium (e.g., web screen).
- Surface Generator: Finally, this component generates the text, assembling retrieved information, creating new sentences, and adding device-specific layout tags.
The system was evaluated through an experiment where participants used both PERCY and a conventional search engine (CMIS website's search engine) to find information on specific CSIRO research topics.
Datasets
The study used the CSIRO website as the source of information. No new datasets were introduced; rather, existing web content was dynamically assembled by PERCY. The evaluation involved 20 university students who acted as research students.
Key Insights and Ideas
- Brochure-like delivery is effective: Users found the PERCY system's output, which mimicked a personalized brochure, to be more engaging and easier to understand than raw search results.
- Coherence and tailoring matter: The combination of presenting information in a structured, coherent manner and tailoring it to the user's profile significantly improved the user experience.
- Beyond just information quantity: While the search engine sometimes provided more raw information, users preferred PERCY's curated and structured delivery, suggesting that quality and presentation are more critical than sheer volume for user satisfaction.
- User models can be modest: The study demonstrated that even a "modest user model" could be sufficient to achieve significant improvements in information delivery.
Key Results
The experiment involved 20 university students who were asked to find information on four CSIRO research topics using both PERCY and the CMIS search engine.
- Post-system questionnaire: While most differences were not statistically significant, PERCY tended to score higher for the coherence of information presentation, particularly in question 7 ("I think the presented information serves well as a useful online brochure"), with a significance level of 0.06. PERCY also scored higher on questions related to meeting user needs and providing sufficient information.
- Exit questionnaire: This questionnaire directly compared the two systems. PERCY was ranked higher by subjects across all questions. Significant differences were found for questions related to attracting attention (p<0.01), providing better explanations (p<0.03), and overall preference for PERCY as an online brochure (p<0.04).
Limitations and Future Work
The authors acknowledge that the experiment was limited by the number and type of subjects, as well as the scope of the study. A key limitation is that while PERCY attracted users' attention better, the study did not definitively prove whether this increased focus translates into further interaction with the corporation or its website. Future work could explore this link between engagement and subsequent user actions.
Summary updated