TREC-6 Interactive Track Report. Part 1: Experimental Procedure and Initial Results

Don Byrd, Russell Swan, James Allan2005 · DOI

TL;DR

This paper introduces a user study for "aspect-oriented retrieval," a new information retrieval task focused on finding documents covering specific facets of a topic, rather than just general relevance. The study evaluated novel system features like an "aspect window" for organizing findings and a 3D visualization for exploring document relationships, aiming to improve user efficiency in complex search scenarios.

What Problem Does This Paper Address?

Traditional information retrieval (IR) systems aim to find documents that are generally relevant to a user's query. However, many real-world search tasks require users to identify and gather information on specific aspects or sub-topics within a broader query. For example, a user researching "alternative energy sources for automobiles" might need to find documents on solar, electric, hydrogen, and biofuel options, as well as information on their environmental impact and cost-effectiveness. The challenge is that standard IR systems don't explicitly support this aspect-oriented approach, leaving users to manually track and organize these distinct pieces of information, which can be time-consuming and error-prone. This paper addresses this gap by designing and evaluating experimental systems and user interfaces specifically for aspect-oriented retrieval.

Methods

The core of this research is a user study designed to evaluate novel system features for aspect-oriented retrieval. The study involved participants searching for information on six predefined topics using either a control system (ZPRISE) or one of two experimental systems (AspInquery and AspInquery Plus). The experimental systems introduced two key features:

  • Aspect Window: A dedicated interface element designed to help users organize documents by aspect. Users could drag documents into this window and assign labels. The system also automatically generated distinguishing terms from the documents within an aspect to aid in labeling.
  • 3D Visualization: AspInquery Plus offered a 3D visualization of retrieved documents, aiming to help users understand the relationships between documents and identify clusters of similar content, which could correspond to different aspects.

The experimental design used a Latin square approach to counterbalance system exposure and minimize learning effects. Participants were also given psychometric tests (verbal fluency and spatial visualization) to explore potential correlations with search performance.

Datasets

The study utilized a corpus of approximately 200,000 newspaper articles from the Financial Times (1991-1994), which is a subset of the larger TREC collection. Six topics were specifically chosen and modified from the TREC-6 adhoc topics for use in this interactive track experiment.

Key Insights and Ideas

  • The Need for Explicit Aspect Support: The study's hypotheses and design underscore the growing recognition that many complex information needs go beyond simple document retrieval and require structured organization of information by sub-topics or "aspects."
  • User Interface as a Key Enabler: The development of specialized tools like the "aspect window" and 3D visualization suggests that effective user interfaces are crucial for supporting novel search paradigms like aspect-oriented retrieval.
  • Potential for 3D Visualization: The paper explores the idea that visualizing document clusters in 3D space could help users identify related documents and potentially discover new aspects or confirm existing ones, based on the cluster hypothesis.
  • Individual Differences Matter: The inclusion of psychometric tests hints at the possibility that users' inherent cognitive abilities (like spatial visualization) might influence their effectiveness with certain interface features.

Key Results

The paper presents the experimental setup and procedure but focuses on the initial design and hypotheses rather than detailed quantitative results of the user study itself. The "Initial Results" mentioned in the title are not elaborated upon in this section of the paper. However, the experimental design involved:

  • 20 participants (8 librarians, 12 general users).
  • Each participant searched six topics.
  • Participants used either a control system (ZPRISE) or one of two experimental systems (AspInquery, AspInquery Plus).
  • The study aimed to test hypotheses such as the helpfulness of an aspect tracking tool and the utility of 3D visualization.

The detailed appendices provide participant demographics and test scores, but the core findings from the user interaction and system performance are not yet presented in this part of the report.

Limitations and Future Work

  • Focus on Procedure: This document is Part 1 of a report, focusing on the experimental procedure and initial setup. The actual results and analysis of the user study are likely to be presented in subsequent parts.
  • Hypothesis Testing: The paper outlines hypotheses but does not present the statistical analysis to confirm or refute them.
  • Limited Scope of Experimental Systems: While the experimental systems introduced novel features, they were based on the Inquery search engine, and the full range of Inquery's capabilities was not necessarily exposed to all participants.
  • Future Work: The paper implies that future work will involve analyzing the data collected from the user study to evaluate the effectiveness of the aspect window and 3D visualization, and to test the stated hypotheses. Further investigation into variations of the visualization to enhance user discovery is also mentioned.

Summary updated

Abstract

The TREC-6 Interactive Track concentrated on Aspect-Oriented Retrieval: finding at least one document that covers each aspect of relevance to a given topic as opposed to the usual information-retrieval task of simply finding documents containing as much information as possible about the topic. For this track, we at the Center for Intelligent Information Retrieval of the University of Massachusetts designed and ran our first-ever user study.
BibTeX
@misc{byrd2005trec,
  author = {Don Byrd and Russell Swan and James Allan},
  title = {{TREC-6 Interactive Track Report. Part 1: Experimental Procedure and Initial Results}},
  year = {2005},
  doi = {10.21236/ada440416},
}
Figure