Use and usability in a digital library search system

Robert K. France, Lucy T. Nowell, Edward A. Fox, Rani A. Saad, Jianxin ZhaoarXiv 1999 · arXiv · OpenReview

TL;DR

This paper chronicles the six-year evolution of MARIAN, a digital library search system, emphasizing how user studies, use pattern analysis, and formative usability evaluations were crucial for transforming it from a technical concept into a widely adopted and useful tool. The authors argue that a deep understanding of users, their behavior, and system usability is essential for creating successful digital library components.

What Problem Does This Paper Address?

The core challenge addressed is how to design and evolve digital library search systems that are not only technically robust but also genuinely usable and useful for a diverse and often "chronic beginner" user base. Digital libraries aim to serve everyone, from casual users to intensive researchers, with varying information needs. Traditional library systems often faced usability issues, and the authors sought to avoid these pitfalls by focusing on user-centric development. They highlight the difficulty of understanding user behavior in the context of digital libraries, where interactions are primarily network-based, making traditional usability studies more complex.

Methods

The paper details a multi-pronged approach to system development and improvement:

  • Formative Usability Evaluation: This involved conducting detailed, extended observation sessions with a representative sample of users. Participants were given benchmark tasks to complete, and their verbal protocols and critical incidents were meticulously recorded. This method helped uncover usability problems and inform design changes.
  • User Studies & Use Pattern Analysis: The authors analyzed system logs to understand how users interacted with MARIAN. This included examining query patterns, result set interactions, and the frequency of different search strategies. This data provided insights into user behavior and guided system evolution.
  • Iterative Design & Evolution: MARIAN's development was not a one-off event but a continuous process. The system evolved over six years, with changes driven by the findings from usability evaluations and log analysis. This iterative approach allowed for refinement of both the interface and underlying functionality.

Datasets

Not discussed in the paper. The paper focuses on the evolution of a single system (MARIAN) using its own usage data and user feedback, rather than external datasets for training or evaluation.

Key Insights and Ideas

  • The "Chronic Beginner" User: Library users often have uneven demand for services, leading to a loss of sophisticated search skills. This necessitates systems that are easy to learn but also offer advanced capabilities.
  • Usability is Not an Afterthought: The paper strongly advocates for integrating usability considerations from the initial design phase through ongoing development, rather than treating it as a post-hoc fix.
  • The Power of Three Pillars: The authors contend that successful digital library systems require a combination of formal usability testing, statistical analysis of system use, and in-depth study of the user community.
  • User Mental Models Matter: Understanding how users conceptualize the search process (their "mental models") is crucial for designing intuitive interfaces.
  • Web Evolution Demands Adaptability: The shift to the World Wide Web required MARIAN to adapt its interface and delivery mechanisms, moving from platform-specific clients to a Web gateway.
  • Simplicity and Consistency: Even with advanced features, users benefit from clear instructions, consistent terminology, and a predictable interface. The move to HTML 2.0 and avoiding complex constructs like frames was a deliberate choice to ensure broad accessibility.

Key Results

  • Significant Usability Improvements: Formative usability evaluations led to profound changes in MARIAN's presentation and functionality, including redesigning query forms and result set displays to be more intuitive and user-friendly. For example, difficulties users had submitting queries were addressed by adding explicit "Do Search" buttons.
  • Increased User Engagement: The iterative improvements, driven by user feedback and analysis, resulted in a "well-subscribed" system, indicating its usefulness and acceptance by the community.
  • Validation of Approximate Matching: User studies confirmed that users preferred approximate matching systems, finding them easier to use and more effective for assembling relevant lists of sources.
  • Query Refinement is Key: Log analysis showed that a significant portion of sessions involved multiple queries, validating the system's design to support query reuse and refinement.
  • Web Interface Success: The WebGate implementation successfully provided much of the power and freedom of the standalone clients, meeting goals for fast loading, minimal page transitions, and intuitive navigation.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that understanding the significance of MARIAN's longer query fields compared to other Web search engines is difficult, with multiple potential contributing factors (user sophistication, system output quality, larger input space).
  • The initial WebGate implementation did not fully meet all goals for query modification and reuse.
  • The authors note that the "Find more like" action was not discovered by most users without prompting during initial usability tests.
  • Future Work:
    • Further usability testing is planned to address issues with the presentation of bibliographic results via hyperlinks, particularly concerning user expectations for author and subject matches.
  • The system is evolving to include personalization features, where the interface can be customized based on user preferences.
  • Future releases aim to support global history across all sessions, allowing users to revisit past queries and results, with ongoing usability considerations for organizing and managing this history.
  • The MARIAN system is being integrated into broader digital library frameworks, including efforts in personalization and a seminal visualization interface.

Summary updated

Abstract

Digital libraries must reach out to users from all walks of life, serving information needs at all levels. To do this, they must attain high standards of usability over an extremely broad audience. This paper details the evolution of one important digital library component as it has grown in functionality and usefulness over several years of use by a live, unrestricted community. Central to its evolution have been user studies, analysis of use patterns, and formative usability evaluation. We extrapolate that all three components are necessary in the production of successful digital library systems.
BibTeX
@misc{france1999use,
  author = {Robert K. France and Lucy T. Nowell and Edward A. Fox and Rani A. Saad and Jianxin Zhao},
  title = {{Use and usability in a digital library search system}},
  year = {1999},
  eprint = {cs/9902013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/cs/9902013},
}
Figure