Abstract
There is broad consensus in the field of Information Retrieval (IR) that search is complex in many use cases and applications, both on the Web and in domain-specific collections. Yet our understanding of complex search tasks, in comparison to simple lookup tasks, is fragmented at best. The Second Workshop on Supporting Complex Search Tasks (SCST 2017) addressed several open research questions: What are the obvious use cases and applications of complex search? What are essential features of work tasks and search tasks to take into account? How does the information-seeking process evolve, and what are the relevant differences between stages? Supporting complex search tasks requires new collaborations across the field of IR. This workshop brought together a diverse group of researchers to discuss search context, process management, and evaluation frameworks. The event featured two keynotes, a poster session with seven contributed papers, and three breakout groups focused on search context, process, and evaluation.
Introduction: Beyond the Search Box
Most commercial search engines are highly optimized for "lookup" tasks—simple, fact-based queries like "What is the capital of France?" However, real-world information seeking is often far more complex. Whether a researcher is conducting a comprehensive literature review or a parent is navigating the safety settings of a new device, these tasks are non-linear, involve heterogeneous sources, and evolve as the user learns.
There is currently a significant gap between the technical capabilities of IR systems and the actual needs of users attempting to accomplish broader "work tasks." The SCST 2017 workshop aimed to bridge this gap by defining, supporting, and evaluating search in these messy, real-world contexts, moving the focus from the query box to the user's ultimate goal.
Keynote Perspectives on Complexity
The workshop opened with two keynotes that set the stage for the day's discussions. Mark Hall addressed the complexity of meaningful interactive IR evaluation, arguing that the line between the "IR task" (the search) and the "work task" (the goal) is disappearing. Evaluation must move beyond simple relevance metrics to determine if a system actually helps a user complete their objective.
Jussi Karlgren discussed the types of search complexity encountered in real-world business applications. He pointed out that seemingly simple questions, such as "Are our customers happy?", require massive, complex curation and aggregation of disparate data sources. Current ad-hoc search models are often ill-equipped to handle this level of data synthesis and long-term task management.
Breakout Group Findings
The core of the workshop involved three specialized breakout groups tasked with developing actionable research blueprints:
1. Search Context and Tasks
This group focused on the challenge of constructing "authentic" tasks for laboratory studies. They proposed using Bloom’s taxonomy of learning outcomes to prompt different search behaviors, moving from simple recall to synthesis and evaluation. A major debate centered on whether it is possible to study complex tasks without their inherent context (device, time pressure, location), as context often defines the complexity itself.
2. Search Process
Participants discussed the need for a "shared task" or collaborative workshop where researchers use diverse methods to solve a single, complex problem. This would allow for a direct comparison of different tools and techniques, highlighting how different systems support various stages of the information-seeking process.
3. Evaluation of Complex Search
Addressing the "reproducibility crisis" in interactive IR, this group proposed an "extract-a-thon." This collaborative effort involves reviewing the methodology sections of published papers to determine if data descriptions are sufficient for reproduction. The group also called for a central repository for interactive track data to standardize user interaction logs.
Innovative Research Highlights
Several contributed papers presented novel approaches to UI and user learning:
- Search User Interface (SUI) Framework: Proposed by Huurdeman, this framework categorizes features into personalizable, informational, and control features across different search stages.
- The Billiard Metaphor: Ventocilla et al. introduced a novel UI for exploring relations in datasets, treating data points like billiard balls to provide an intuitive, exploratory faceted search experience.
- Concept Mapping: Research by Egusa et al. demonstrated that "concept maps" created by users before and after searching are a viable metric for measuring how much a user has learned during a complex search session.
Challenges and Future Directions
A primary limitation identified by the workshop is the lack of a methodical approach for constructing authentic complex tasks. Most current research relies on simulated tasks that may not capture the urgency or evolution of real-world needs.
To move forward, the community must develop a "check-list and quality certification" for data characterization to improve research reproducibility. Furthermore, organizing domain-specific workshops (e.g., for legal or medical search) with real-world experts will be essential for defining authentic tasks and moving toward comparable, rather than just identical, data sets across the field.
Summary updated