PINGAN_AI at SemEval-2022 Task 9: Recipe knowledge enhanced model applied in Competence-based Multimodal Question Answering

Zhihao Ruan, Xiaolong Hou, Lianxin JiangNAACL 2022

TL;DR

This paper introduces a knowledge-enhanced BERT model for recipe-based question answering (QA) that significantly improves performance by incorporating auxiliary recipe knowledge and entity embeddings. The model achieved a 3rd place ranking in the SemEval-2022 Task 9, demonstrating the effectiveness of their approach in understanding complex recipe semantics.

What Problem Does This Paper Address?

The SemEval-2022 Task 9, "Competence-based Multimodal Question Answering," focuses on answering questions about cooking recipes, which often involve rich semantic annotations and aligned text-video data. A key challenge is that many answers cannot be directly extracted from the original recipe text. Instead, the information is often embedded within structured knowledge annotations, such as cooking role labeling (CRL) and semantic role labeling (SRL), or requires understanding implicit relationships like co-references and hidden roles. Traditional QA models, even powerful ones like BERT, struggle with these implicit knowledge dependencies. This paper aims to bridge this gap by developing a model that effectively leverages this external recipe knowledge to improve QA performance.

Methods

The core of the proposed system is a BERT-based model enhanced with two primary knowledge integration strategies:

  • Knowledge Auxiliary Text: This method augments the original recipe text with additional information derived from CRL annotations. Specifically, it incorporates:
  • Co-reference knowledge: Links different mentions of the same entity or concept, allowing the model to resolve ambiguities and track information across the text. For example, if "mixture" is mentioned, and the co-reference annotation specifies it refers to "small balls," the model receives "mixture (small balls)" as input.
  • Hidden-role knowledge: Supplements action tokens with information about the roles of involved entities, inspired by prompt learning. For instance, if the action is "use" and the hidden role is "TOOL: knife," the model might receive "by using a knife" as auxiliary text.
  • Knowledge Embedding: This technique encodes structured knowledge into dense vector representations. The model integrates:
  • Universal Part-of-Speech (u-pos) knowledge: Embeddings for the grammatical role of words.
  • Entity knowledge: Embeddings for the identified entities (e.g., ingredients, tools) within the recipe.

These knowledge embeddings are integrated into the BERT model in two ways: either by adding them to the token embedding layer (Plan A) or by incorporating them into the model's header (Plan B).

Additionally, the authors designed an answer extraction task pipeline. This pipeline involves an extraction-based model, an automatic keyword labeling module, and an answer generation module. During training, keywords from answers are identified and labeled in the recipe text. During prediction, the model outputs keywords, which are then used to construct the final answer. For certain QA categories (6 out of 18), a rule-based method is employed, particularly for tasks like counting.

Datasets

The paper uses the R2VQ (Recipe Reading and Video Question Answering) dataset, provided by the SemEval-2022 Task 9 organizers. This dataset consists of cooking recipes and associated video screenshots. It contains 1000 samples, split into training (800), validation (100), and testing (100) sets. The recipe text includes annotations for Cooking Role Labeling (CRL) and Semantic Role Labeling (SRL). The model specifically uses the "directions" part of the recipes as input.

Key Insights and Ideas

  • Implicit Knowledge is Crucial: The paper strongly emphasizes that a significant portion of answers in recipe QA tasks are not directly stated but are inferable from structured knowledge annotations. This highlights the need for models that can go beyond simple text extraction.
  • Auxiliary Text Improves Context: Augmenting the original recipe text with co-reference and hidden-role information (knowledge auxiliary text) provides richer context, helping the model understand implicit relationships and disambiguate entities.
  • Knowledge Embeddings Enhance Understanding: Encoding entity and u-pos information into embeddings allows the model to leverage structured knowledge about ingredients, tools, and their grammatical roles, leading to better answer prediction.
  • Embedding Layer Integration is Key: Integrating knowledge embeddings directly into the BERT embedding layer (Plan A) yielded better performance than incorporating them into the model's header (Plan B). This suggests that early integration of structured knowledge is more beneficial.
  • Hybrid Approach is Effective: Combining a deep-learning model with a rule-based module for specific QA patterns (like counting tasks) allows the system to leverage the strengths of both approaches, leading to improved overall performance.

Key Results

The proposed knowledge-enhanced BERT model achieved a 3rd place ranking in the SemEval-2022 Task 9.

  • The system achieved a word-level F1 score of 82.62 and an exact match score of 78.21.
  • Ablation studies showed significant improvements from the knowledge-enhanced methods:
  • The baseline BERT model (without knowledge enhancement) achieved a squad word-level F1 score of 61.
  • Using only the knowledge auxiliary text method resulted in a word-level F1 score of 88.78.
  • Using both knowledge auxiliary text and knowledge embedding methods achieved the best score of 89.23.

Limitations and Future Work

  • Rule-based Module Focus: While the paper mentions a hybrid approach, the detailed discussion and experimental results primarily focus on the deep-learning model module and its knowledge enhancement. The performance contribution of the rule-based module is not deeply analyzed.
  • Header Integration Issues: The performance dropped when knowledge embeddings were placed in the model's header. The authors suggest this might be due to inappropriate parameter settings or a too-simple header design.
  • Future Work: The authors plan to incorporate video data into their model, which is a significant multimodal aspect of the R2VQ dataset. They also aim to extend their approach to predict answers for all pattern categories using a single model, rather than relying on a hybrid rule-based and deep-learning approach.

Summary updated

BibTeX
@inproceedings{ruan2022pingan,
  author = {Zhihao Ruan and Xiaolong Hou and Lianxin Jiang},
  title = {{PINGAN\_AI at SemEval-2022 Task 9: Recipe knowledge enhanced model applied in Competence-based Multimodal Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure