TL;DR
This paper tackles the challenge of understanding procedural texts beyond the well-studied cooking domain. They propose a framework for predicting "flow graphs" (FG) from open-domain procedural texts and introduce the wikiHow-FG corpus to enable this. Their work demonstrates that domain adaptation from cooking data significantly improves FG prediction performance in new domains, especially when data is scarce.
What Problem Does This Paper Address?
Procedural texts, like recipes or assembly instructions, are crucial for guiding humans through tasks. To automate or better understand these tasks, we need to represent them in a structured, actionable format. Flow graphs (FG) are one such representation, capturing the sequence of actions and relationships between entities. While previous work has successfully applied FG prediction to cooking recipes, this approach hasn't been generalized to other domains. This is a significant limitation because creating large, annotated datasets for new domains is labor-intensive and expensive. The authors aim to bridge this gap by developing a method that can predict FGs for procedural texts from various domains, even with limited target-domain data, by leveraging existing cooking domain knowledge.
Methods
The core of the paper's contribution lies in adapting an existing FG prediction framework to open-domain procedural texts. Their framework operates in two stages:
- Node Prediction: This stage identifies entities within the text and assigns them specific tags (e.g., action, component, tool). They frame this as a sequence labeling problem, similar to Named Entity Recognition (NER), and utilize a pre-trained DeBERTa model for this task.
- Edge Prediction: Once nodes are identified, this stage predicts the relationships (labeled edges) between them, forming a directed acyclic graph. They employ a biaffine dependency parser for this, which models edge prediction and label prediction separately.
A key aspect of their methodology is domain adaptation. They train models on a large cooking domain dataset (English-FG) and then fine-tune them on a smaller dataset from a target domain (wikiHow-FG). This approach is designed to overcome the challenge of limited annotated data in new domains. They also explore two data augmentation techniques: step swapping (reordering causally independent steps) and word replacement (substituting words with synonyms) to further boost performance in low-resource settings.
Datasets
- English-FG Corpus: This is an existing corpus of English recipe flow graphs, used as the source domain for training.
- wikiHow-FG Corpus: This is a newly introduced corpus by the authors. It's constructed from wikiHow articles, a popular source of how-to guides. The corpus covers four domains: Food and Entertaining, Hobbies and Crafts, Home and Garden, and Cars & Other Vehicles. For each domain, 30 articles were annotated, resulting in a total of 120 articles. This corpus is designed to represent procedural texts in non-cooking domains.
Key Insights and Ideas
- Domain Adaptation is Crucial: The most significant finding is that domain adaptation from cooking to other procedural domains is highly effective, especially when target domain data is scarce. This suggests that the underlying structure of procedural texts, even across different domains, shares enough commonalities to allow knowledge transfer.
- Cooking Domain as a Strong Source: The cooking domain, with its rich resources, serves as a valuable source for pre-training FG prediction models, enabling better performance in less data-rich domains.
- Step Swapping Augmentation Helps (Sometimes): The step-swapping data augmentation technique showed a slight improvement in some domains, particularly for node prediction. This highlights the potential of intelligently generating synthetic data to improve model robustness.
- Edge Prediction is More Data-Hungry: The results indicate that edge prediction is more sensitive to the amount of training data compared to node prediction, requiring more examples to achieve comparable performance.
- Tag-Level Domain Specificity: While domain adaptation helps overall, the effectiveness varies by tag. Tags like "Ae" (Action by expert) are more domain-agnostic, while "C" (Component) and "T" (Tool) show greater benefits from domain adaptation, suggesting they are more domain-specific.
Key Results
- Domain Adaptation Outperforms: Domain-adaptation models consistently achieved higher F1 scores than models trained solely on the cooking-only or target-only data across all tested domains. For instance, in Hobbies and Crafts, domain adaptation improved F1 scores by 9.5% for node prediction.
- Edge Prediction Performance: Even with only the English-FG corpus, edge prediction achieved F1 scores of 58.7% or more in non-cooking domains.
- Pipeline Performance Drop: When using predicted nodes for edge prediction (a more realistic scenario), performance dropped significantly, especially in domains further from cooking (around 24.6% drop). This underscores the importance of accurate node prediction for the overall pipeline.
- Augmentation Impact: Step-swapping augmentation provided a slight improvement in some node prediction tasks. Word replacement, however, did not consistently improve performance, possibly due to mismatches between replaced words and their intended roles.
Limitations and Future Work
- Pipeline Errors: The significant performance drop in the pipeline experiment (using predicted nodes for edge prediction) highlights a key limitation. Errors in node prediction directly impact edge prediction, suggesting that improving node prediction accuracy is critical.
- Data Augmentation Nuances: While step swapping showed some promise, word replacement was less effective. The authors suggest that more sophisticated data augmentation techniques might be needed.
- Future Directions: The authors propose extending their framework to other domains like material science and biochemistry. They also suggest exploring more advanced data augmentation strategies to further enhance performance. Additionally, they note that exploiting visual information from wikiHow articles, which was not used in this study, could be a promising avenue for future research.
Summary updated