An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites

Cécile Paris, Mingfang Wu, Anne-Marie Vercoustre, Stephen Wan, Peter Wilkins, Ross Wilkinson2003

TL;DR

This paper introduces PERCY, a system that delivers tailored and coherent information from corporate websites, akin to a personalized brochure. An empirical study found that users preferred PERCY's output over traditional search engine results, indicating that this approach is more engaging and understandable for users seeking information.

What Problem Does This Paper Address?

Navigating corporate websites to find specific information can be challenging. Traditional methods like searching and browsing often yield overwhelming amounts of data that aren't tailored to individual user needs or presented in an easily digestible format. This makes it difficult for users to quickly grasp relevant information and for corporations to effectively communicate their offerings. The authors aim to address this by exploring whether a system that combines user modeling with coherent, tailored document delivery can provide a superior user experience compared to standard search engines.

Methods

The core of the paper is the PERCY system, which integrates information retrieval with user modeling and discourse-oriented technologies. PERCY takes a user's query and a modest user profile (including stereotypical information like job position and individual information like interests) to construct a coherent, brochure-like document. This involves:

  • Virtual Document Planner (VDP): This component creates a "discourse tree" that dictates the information to be included and its organization. It uses metadata to pull relevant snippets from existing web pages.
  • Discourse Planner: Modeled on existing work, this planner uses a library of "discourse rules" to ensure the information is presented logically and coherently, mimicking how information is structured in traditional brochures.
  • Presentation Planner: This module decides on the amount of information to present and tailors the presentation to the chosen delivery medium (e.g., web screen).
  • Surface Generator: Finally, this component generates the text, assembling retrieved information, creating new sentences, and adding device-specific layout tags.

The system was evaluated through an experiment where participants used both PERCY and a conventional search engine (CMIS website's search engine) to find information on specific CSIRO research topics.

Datasets

The study used the CSIRO website as the source of information. No new datasets were introduced; rather, existing web content was dynamically assembled by PERCY. The evaluation involved 20 university students who acted as research students.

Key Insights and Ideas

  • Brochure-like delivery is effective: Users found the PERCY system's output, which mimicked a personalized brochure, to be more engaging and easier to understand than raw search results.
  • Coherence and tailoring matter: The combination of presenting information in a structured, coherent manner and tailoring it to the user's profile significantly improved the user experience.
  • Beyond just information quantity: While the search engine sometimes provided more raw information, users preferred PERCY's curated and structured delivery, suggesting that quality and presentation are more critical than sheer volume for user satisfaction.
  • User models can be modest: The study demonstrated that even a "modest user model" could be sufficient to achieve significant improvements in information delivery.

Key Results

The experiment involved 20 university students who were asked to find information on four CSIRO research topics using both PERCY and the CMIS search engine.

  • Post-system questionnaire: While most differences were not statistically significant, PERCY tended to score higher for the coherence of information presentation, particularly in question 7 ("I think the presented information serves well as a useful online brochure"), with a significance level of 0.06. PERCY also scored higher on questions related to meeting user needs and providing sufficient information.
  • Exit questionnaire: This questionnaire directly compared the two systems. PERCY was ranked higher by subjects across all questions. Significant differences were found for questions related to attracting attention (p<0.01), providing better explanations (p<0.03), and overall preference for PERCY as an online brochure (p<0.04).

Limitations and Future Work

The authors acknowledge that the experiment was limited by the number and type of subjects, as well as the scope of the study. A key limitation is that while PERCY attracted users' attention better, the study did not definitively prove whether this increased focus translates into further interaction with the corporation or its website. Future work could explore this link between engagement and subsequent user actions.

Summary updated

Abstract

Increasingly, people find information about corporations through their web sites, as opposed to reading printed brochures. Typically, the only tools they have to find that information is searching and browsing. But can we do better with a modest user model and coherent delivery? Can we provide, through the web, more relevant and coherent information about a corporation, information specially tailored to specific clients and presented in a useful and understandable manner? Our conjecture was that a tailored hypermedia system offering tailored and coherent information would indeed provide a good alternative to the current search-and-browse facilities, also allowing corporations to be more responsive to their clients&amp;apos; needs. In this paper, we present a preliminary study aimed at testing this hypothesis. We designed an experiment aimed at investigating the effect of delivering coherent and tailored information in response to user&amp;apos;s query, in comparison to delivering the results of conventional search engines. We employed our experimental system, PERCY, which exploits information retrieval, user modeling and discourse-oriented technologies to retrieve and select parts of existing documents from CSIRO website and assemble them back into a coherent and tailored webpage. Our experiment consisted of asking users to evaluate and compare the results produced by PERCY and those produced by a typical search engine, given a query about the some project or research topic at CSIRO. Our experimental results indicate that users preferred the output of Percy, as it attracted more their attention and they understood the results better. Keywords: information retrieval, discourse model, evaluation, tailored delivery, coherent delivery
BibTeX
@misc{paris2003empirical,
  author = {Cécile Paris and Mingfang Wu and Anne-Marie Vercoustre and Stephen Wan and Peter Wilkins and Ross Wilkinson},
  title = {{An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites}},
  year = {2003},
}
Figure