Determining Event Outcomes: The Case of #fail

Srikala Murugan, Dhivya Chinnappa, Eduardo BlancoEMNLP 2020 · DOI

TL;DR

This paper moves beyond simply detecting if an event happened to determining its actual outcome. By analyzing 4,000 tweets tagged with #cookingFail or #bakingFail, researchers demonstrate that while social media users often label their efforts as "failures," a multimodal approach—combining text and image analysis—reveals that many of these outcomes are actually edible, albeit visually imperfect.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), researchers have spent significant effort studying event factuality (did the event occur?) and veridicality (is the statement true?). However, even if an event definitely happened, the result is often unknown. Did a phone call result in a clear conversation? Did a planted tree eventually bear fruit?

The authors argue that event outcomes are a matter of degree rather than a binary "success" or "failure." They chose the domain of cooking and baking because these events have clear goals (edibility and quality) and are frequently documented on social media. The challenge lies in the fact that social media language is often hyperbolic or sarcastic; a user might tweet "#bakingFail" even if they produced a delicious, if unattractive, cake.

Methods

The researchers developed a multimodal neural network to predict two target variables: edibility (Yes/No) and quality (As Expected, Partial Success, Alternative, or Unknown).

The multimodal neural network architecture combining text embeddings, visual features, and semantic image tags.

The architecture consists of three main components:

  1. Text Component: A Long Short-Term Memory (LSTM) network that processes tweet text using pre-trained GloVe embeddings.
  2. Image Feature Component: A pre-trained InceptionNet model used to extract high-level visual features from the images.
  3. Image Tag Component: To bridge the gap between pixels and semantics, the Google Cloud Vision API generated descriptive tags (e.g., "cake," "fondant"). These tags were then fed into a separate LSTM.

The outputs are concatenated and passed through a Softmax layer for final classification.

Datasets

The authors introduced a new, manually annotated corpus of 4,000 English tweets (2,000 #cookingFail and 2,000 #bakingFail). The set was split evenly between text-only tweets and those containing both text and an image.

Examples of annotated tweets showing the variety of outcomes and labels in the #fail dataset.

Two graduate students annotated the tweets for relevancy, edibility, and quality. They found that 59.7% of "fail" tweets with images actually resulted in something edible, highlighting the gap between user sentiment and functional outcome.

Key Insights and Ideas

  • The "Fail" is Often Visual, Not Functional: The #fail hashtag is frequently used to describe "ugly" food that is perfectly fine to eat. In tweets with images, nearly 60% were edible.
  • Images Provide the "Ground Truth": Social media text is unreliable due to sarcasm. Human annotators agreed much more often when an image was present, and machine learning models performed significantly better when they could "see" the result.
  • Edibility is Easier than Quality: It is much simpler for a model to determine if a dish is edible than to categorize the type of failure. Predicting "quality" remains a challenge because it requires understanding the cook's original intent versus the final result.

Key Results

  • Edibility Prediction: The multimodal network (text + images) achieved an F1-score of 0.70, a significant improvement over the text-only model's 0.54.
  • Quality Prediction: This task proved harder. Both text-only and multimodal models hovered around an F1-score of 0.53–0.54, barely outperforming the majority baseline.
  • Ablation Studies: The full network (text + InceptionNet + Vision API tags) performed better than any single component, proving that semantic tags and raw visual features provide complementary information.

Limitations and Future Work

Error analysis showed that many mistakes occurred because the model lacked "common sense" cooking knowledge—for example, not knowing that a "hard-boiled egg" cannot be used to make "banana bread." Additionally, "quality" labels are subjective; if a user tries to make an omelet but ends up with scrambled eggs, the model struggles to distinguish between a "failure" and an "alternative success."

Summary updated

BibTeX
@inproceedings{murugan2020determining,
  author = {Srikala Murugan and Dhivya Chinnappa and Eduardo Blanco},
  title = {{Determining Event Outcomes: The Case of \#fail}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.359},
}
Figure