TL;DR
This paper presents a winning hybrid question-answering system for a cooking recipe QA task that requires understanding semantic roles. Their approach combines rule-based systems, a Transformer model, and a neural classifier, achieving a high exact match score by effectively identifying question intent and extracting relevant information from recipe annotations.
What Problem Does This Paper Address?
The paper tackles the challenge of building a question-answering (QA) system capable of understanding real-world domains, specifically cooking recipes. This task, known as R2VQ (Recipe-based Question Answering), goes beyond simple information retrieval. It requires systems to demonstrate "competence-based comprehension," meaning they need to reason about actions, ingredients, locations, and other semantic relationships within a recipe, much like a human would. Existing QA systems often struggle with this level of nuanced understanding, especially when dealing with implicit information and complex reasoning. The task also involves questions categorized into "question families" that test specific reasoning competencies, with a significant portion of questions potentially having "no answer" (N/A).
Methods
The SRPOL system employs a hybrid approach, combining several components:
- Rule-Based System: This forms the core of the system, focusing on:
- Intent Identification: Using regular expressions and syntactic/lexical analysis to categorize questions into one of 17 predefined types (e.g., counting, location, method).
- Data Extraction: Leveraging semantic role labeling (SRL) and cooking role labeling (CRL) annotations from the recipe dataset to find relevant information.
- Response Generation: Constructing answers based on the identified intent and extracted data, sometimes requiring rephrasing or aggregation.
- Question Answering Transformer (RC System): For cases where the rule-based system doesn't yield an answer or has low confidence, a Transformer-based Reading Comprehension (RC) model (specifically ELECTRA) is used. This model is fine-tuned on the task's recipes.
- N/A Classifier: A fine-tuned BERT-based multilingual model is used to identify questions that genuinely have no answer within the recipe, acting as a fallback mechanism.
The system prioritizes the rule-based approach, using the RC system and N/A classifier as fallbacks or for specific categories where they provide a boost.
Datasets
- R2VQ Dataset: The primary dataset used for training, validation, and testing. It consists of question-answer pairs derived from cooking recipes. The dataset is characterized by extensive annotations, including semantic role labeling (SRL) and cooking role labeling (CRL).
- Training: 26,526 questions
- Validation: 3,829 questions
- Test: 3,442 questions
- SQuAD Dataset: Used for fine-tuning the Transformer-based RC models (BERT, RoBERTa, ELECTRA).
Key Insights and Ideas
- Semantic Roles are Crucial: The success of the system heavily relies on the rich semantic annotations (SRL and CRL) provided with the recipes. These annotations are key to understanding the relationships between verbs, objects, and other entities.
- Hybrid Approach Outperforms Single Models: Combining a robust rule-based system with powerful deep learning models (Transformers) and a dedicated N/A classifier proved more effective than any single approach alone.
- Intent Identification is the Bottleneck: The authors identify correct intent identification as the most challenging aspect, particularly for categories like "METHOD" which have multiple sub-intents.
- Images are Less Useful Than Text: Despite the multimodal nature of the task (recipes often include images), the authors found that images provided little helpful information for answering questions, with only a small fraction being relevant. This led them to focus solely on the textual content.
- Human Performance is Lower Than Expected (on EM): A human benchmark revealed a significantly lower Exact Match (EM) score (52%) compared to the system's (92.53%). This is attributed to the strictness of the EM metric, which penalizes semantically correct but lexically different answers, and the human tendency towards more creative or varied responses.
Key Results
- Overall Exact Match (EM) Score: The system achieved an impressive 92.53% EM on the test set.
- Category-Specific Performance: EM scores varied by category, with some (e.g., EXTENT, LOCATION CHANGE) reaching 100% during training/validation, while others (e.g., COUNTING TIMES, METHOD) were lower.
- RC Model Performance: ELECTRA achieved the best EM score among tested RC models at 31.0% on the test set, indicating the limitations of purely extractive QA for this task.
- N/A Classifier Performance: The N/A classifier achieved an F1 score of 82.7% when using the full recipe passage (ingredients and directions), highlighting the importance of ingredient information.
- Post-Evaluation Improvement: Further refinements to the rule-based system led to a final EM score of 92.969%.
Limitations and Future Work
- Intent Identification Challenges: The system still struggles with accurately identifying intents for certain categories, particularly "METHOD," which has multiple sub-intents.
- Non-Extractive Answers: The task requires generating answers that go beyond simply extracting text spans, involving rephrasing, aggregation, and counting, which are complex for current models.
- Annotation Automation: A significant future direction is the automation of SRL/CRL annotation for new domains, which is currently a manual and labor-intensive process.
- Generative Models: The authors suggest that incorporating generative models could improve the system's ability to produce more natural and varied answers.
- Domain Applicability: While the system showed some applicability to other domains (makeup, furniture, crafts) without modification, further research is needed to adapt it more broadly.
- Question Diversity: The authors identified that the competition dataset might be missing certain question types, such as those asking about ingredient amounts, types, or simple yes/no questions.
Summary updated