Action Recognition and State Change Prediction in a Recipe Understanding Task Using a Lightweight Neural Network Model (Student Abstract)

Qing Wan, Yoonsuck ChoeAAAI 2020 · DOI

TL;DR

Researchers have developed a lightweight neural network that outperforms complex "simulator" models at understanding recipe instructions. By using a simplified GRU-based architecture and a novel tangent-based loss function, the model achieved a 12% improvement in predicting ingredient state changes while requiring 85% less training data.

What Problem Does This Paper Address?

Understanding procedural language—like the steps in a cookbook—is a significant challenge in Natural Language Processing (NLP). To correctly follow a recipe, a system must not only recognize the action (e.g., "whisk," "sauté") but also predict the resulting state change of the entities involved (e.g., the eggs become "foamy," the onions become "translucent").

Previous state-of-the-art approaches attempted to solve this by building explicit "simulator" modules that modeled the physics of the actions. However, these models are often unnecessarily complex, computationally expensive, and require massive amounts of labeled data to train effectively. The authors of this paper set out to prove that a much simpler, "lightweight" architecture could achieve better results by focusing on the mathematical coupling of action and state prediction.

Methods

The authors moved away from complex simulation and instead used a streamlined Encoder-Decoder architecture:

  • Encoder: A Dynamic Recurrent Neural Network (RNN) using Gated Recurrent Units (GRU).
  • Decoder/Predictor: A Multi-Layer Perceptron (MLP) that branches into two tasks: action recognition and state change prediction.
  • Input Representation: Interestingly, the model eschews modern word embeddings (like Word2Vec or BERT) in favor of simple one-hot encoding. This emphasizes the model's "lightweight" nature and its reliance on structural learning rather than pre-trained semantic knowledge.

The core innovation is a novel tangent-based loss function. Instead of standard Mean Squared Error or Cross-Entropy, they designed a loss function based on the tangent of the difference between the label ($Y$) and the prediction ($P$): $$l(\mathbf{Y}, \mathbf{P}) = \sum_{i=1}^{m} 10 \times \tan\left(\frac{\pi}{2} |\mathbf{Y}(i) - \mathbf{P}(i)|\right)$$ Because the tangent function approaches infinity as the argument approaches $\pi/2$, it provides an increasingly steep penalty for incorrect predictions, which the authors argue helps the model converge more effectively on the specific causal relationships in procedural text.

Datasets

The study utilized the dataset introduced by Bosselut et al. (2018):

  • Total Pool: 120,000 recipe samples.
  • Training Set: While the original baseline used over 65,000 samples, this model was trained on a randomly selected subset of only 10,000 samples (9k for training, 1k for validation).
  • Test Set: A standard benchmark of 693 samples.

Key Insights and Ideas

  • Simplicity Over Simulation: The results suggest that explicit action simulators might be "overkill" for recipe understanding. A well-tuned, simple neural network can capture the underlying logic of state changes just as well, if not better.
  • The Power of the Loss Function: The "aha moment" here is the use of the tangent function. By exploiting the mathematical properties of $\tan(x)$—specifically its convexity and rapid growth—the authors created a more sensitive "error signal" that allows the model to learn from much smaller datasets.
  • Data Efficiency: The model's ability to outperform the baseline using only ~15% of the training data is a strong argument for "small data" approaches in specialized NLP tasks.

Key Results

The lightweight model demonstrated significant performance gains over the previous state-of-the-art (SOTA):

  • State Change Accuracy: Achieved 67% accuracy, a substantial jump from the 55% reported by Bosselut et al. (2018).
  • Action Recognition: The model achieved ~81% accuracy (a metric the previous SOTA did not explicitly report).
  • Resource Efficiency: The model reached these benchmarks using significantly fewer training samples (10K vs. 65K+), proving that architectural simplicity can lead to better generalization.

Limitations and Future Work

As a "Student Abstract," the paper is concise, leaving some areas for further exploration:

  • Vocabulary Constraints: By using one-hot encoding instead of embeddings, the model might struggle with "out-of-vocabulary" words or synonyms not present in the small 10K training set.
  • Evaluation Tolerance: The authors used a "one missing tolerance" rule (allowing one item to be missing in a prediction), which is consistent with previous benchmarks but may overlook finer-grained errors.
  • Scaling: Future work could involve testing this tangent-based loss function on more complex procedural tasks outside of the culinary domain, such as chemistry experiments or assembly manuals.

Summary updated

BibTeX
@inproceedings{wan2020action,
  author = {Qing Wan and Yoonsuck Choe},
  title = {{Action Recognition and State Change Prediction in a Recipe Understanding Task Using a Lightweight Neural Network Model (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7245},
}
Figure