TL;DR
This paper introduces a new challenge: procedural document retrieval, which goes beyond simple keyword matching to understand the sequence of steps in instructions. They propose GFPDR, a novel retrieval model that leverages procedural graphs to capture this crucial sequence structure, achieving state-of-the-art results on their newly created dataset.
What Problem Does This Paper Address?
When people encounter errors while performing tasks like cooking or assembling furniture, they often turn to procedural documents (cookbooks, manuals) for help. The challenge is to retrieve the most relevant document that not only contains the right information but also aligns with the user's sequence of actions and the execution flow of the task. Existing retrieval methods, which primarily focus on declarative knowledge (facts, entities), often overlook this critical sequential structure. This leads to less effective solutions and requires more modifications. The authors highlight that even with similar step content, different execution sequences can drastically alter the outcome, making sequence structure a vital factor for accurate retrieval.
Methods
The core of the proposed solution is the Graph-Fusion Procedural Document Retriever (GFPDR). It tackles the problem in three main parts:
- Procedural Graph Construction: Documents are transformed into procedural graphs. These are directed weighted graphs where nodes represent actions, constraints, or gateways (AND, OR, LOOP), and edges represent dependencies (sequence flow, condition flow). This graph structure explicitly models the sequence and relationships between steps.
- Edge-Aware Graph Attention Module (EAGAT): This module processes the procedural graphs. It uses graph attention to capture the sequence structure and dependencies within the graph, updating node embeddings based on their neighbors and edge features.
- Graph-Fusion Document Encoder: This component encodes the procedural document itself and then fuses its representations with the features extracted from the procedural graph. This fusion aims to bridge the gap between the document's textual content and the structural information from the graph, creating a comprehensive representation. A late-interaction architecture is used for efficiency, allowing document encoding offline.
Datasets
The authors introduce a new dataset called Procedural Questions. This dataset consists of:
- Queries: These are generated based on common failure types (missing steps, swapped steps, wrong details) in procedural tasks. They are derived from existing how-to article corpora and augmented with LLM-generated erroneous operations.
- Documents: These are procedural documents, often from sources like WikiHow, that serve as the ground truth for relevant answers.
The dataset was manually curated and reviewed to ensure quality and consistency.
Key Insights and Ideas
- Sequence Matters: The paper strongly emphasizes that for procedural tasks, the order and structure of steps are as crucial as the content itself. Ignoring this leads to suboptimal retrieval.
- Procedural Graphs as a Solution: Representing procedural documents as graphs is an effective way to capture their inherent sequential structure and dependencies.
- Fusion is Key: Combining textual document representations with graph-based structural information (via GFPDR) leads to superior retrieval performance compared to methods that rely solely on text.
- LLMs for Data Generation: LLMs can be effectively used to generate realistic failure scenarios and erroneous operations, significantly reducing the manual effort required for dataset creation.
- Zero-Shot LLM Rerankers Struggle: Surprisingly, state-of-the-art zero-shot LLM-based rerankers performed poorly on this specific task, suggesting that current LLM capabilities may not fully grasp the nuances of procedural sequence retrieval without fine-tuning or specialized architectures.
Key Results
- State-of-the-Art Performance: GFPDR significantly outperforms 14 baseline models across various architectures (sparse retrievers, bi-encoders, cross-encoders, generative models) on the Procedural Questions dataset. It achieves the best performance across all metrics, including Hit@1, Hit@5, NDCG@3, and MRR@10.
- Significant Improvement: GFPDR shows a notable improvement of +10.86% in Hit@1 compared to the strongest baseline (ColBERTv2), demonstrating its superior ability to distinguish between similar procedural documents by leveraging sequence structure.
- Ablation Studies: Removing key components of GFPDR (like the Graph-Document Match Loss or the Edge-Aware Graph Attention Module) led to performance drops, validating the effectiveness of each module.
- Efficiency: GFPDR uses a late-interaction design, allowing for efficient offline document encoding and comparable online query time complexity to models like ColBERTv2.
Limitations and Future Work
- Graph Construction Noise: The procedural graphs constructed by LLMs can contain noise (e.g., nodes representing multiple actions), which can impact retrieval. Future work could focus on more robust graph construction methods.
- False Negatives: Like many datasets, Procedural Questions might contain false negatives (documents with similar content but different titles). Further evaluation is needed to fully understand their impact.
- Corpus Diversity: The current dataset primarily uses WikiHow-style documents. Expanding to more concise sources (like Allrecipes) would test generalization better.
- LLM Integration: While LLM-based rerankers performed poorly, future work could explore how to effectively integrate procedural graphs with LLMs to enable rerankers to better capture sequence structures.
Summary updated