Madly Ambiguous: A Game for Learning about Structural Ambiguity and Why It's Hard for Computers

Ajda Gokcen, Ethan Hill, Michael WhiteNAACL 2018 · DOI

Introduction

"Madly Ambiguous" is an open-source educational game designed to demystify the complexities of natural language processing (NLP) for the general public. Specifically, it focuses on structural ambiguity, a phenomenon where a single sentence can have multiple grammatical interpretations. A classic example is Prepositional Phrase (PP) attachment: in the sentence "Jane ate spaghetti with a fork," the phrase "with a fork" describes the tool used for eating. However, in "Jane ate spaghetti with meatballs," the same syntactic structure describes an ingredient.

While humans use world knowledge to resolve these ambiguities instantly, computers often struggle to distinguish between semantic roles like UTENSIL, PART, MANNER, or COMPANY. The authors aim to show why this remains a "hard" problem for AI, even as models become more sophisticated.

Methods: Rule-Based vs. Neural Approaches

The game challenges users to "trick" a computer character, Mr. Computer Head, using two distinct NLP backends to classify input into four semantic roles:

  1. Basic Mode (Rule-Based): This mode represents "classic" NLP. It utilizes Part-of-Speech (POS) tagging and lemmatization to identify the head word of a phrase and then queries WordNet, a curated lexical database, to trace its hypernyms. If a word like "spoon" is found under the "utensil" branch, it is classified accordingly.
  2. Advanced Mode (Neural/Embeddings): Representing modern NLP, this mode uses word2vec embeddings pre-trained on the Google News corpus. The system maps user input into a high-dimensional vector space and assigns a category based on the nearest cluster of training examples.
Visualization of word clusters in the Advanced Mode embedding space showing semantic proximity and potential polysemy traps.

Adversarial Pedagogy and Key Insights

The core of the game's educational value lies in its "adversarial" design. By framing the interaction as a challenge to fool the AI, the game prompts users to explore the boundaries of language and logic.

The game interface where users input noun phrases to challenge the AI's classification logic.

Several key insights emerged from user interactions and model performance:

  • The Polysemy Trap: Embedding models often fail when words have multiple meanings. For example, "relish" as a condiment (PART) might be mathematically closer to "relish" as an emotion (MANNER) in vector space, leading to misclassification.
  • Lack of Common Sense: Neither model possesses true world knowledge. A rule-based system might misidentify a person as a musical instrument if a database contains an archaic definition (e.g., WordNet listing "trump" as an archaic term for "trumpet").
  • Visual Grounding: A major design insight was that users grasp linguistic concepts much faster when interpretations are paired with humorous, literal illustrations (e.g., a person trying to eat spaghetti using a meatball as a utensil).

Results and Impact

Since its launch in 2017, the game has collected over 13,000 user inputs. Despite users actively trying to subvert the system, the AI performed respectably:

  • Basic Mode Accuracy: 64%
  • Advanced Mode Accuracy: 70%

Both modes significantly outperformed a random baseline of 29%. More importantly, the game has proven effective in diverse settings, from science museums to undergraduate classrooms, successfully conveying that AI "understanding" is often a matter of statistical proximity or rigid hierarchies rather than genuine human-like comprehension.

Future Directions

Future work aims to expand the game's scope beyond PP attachment to other forms of ambiguity, such as coordination ambiguity (e.g., "the old dogs and cats"). Additionally, the authors propose using compositional character models to better handle out-of-vocabulary (OOV) words and implementing dynamic explanations to help users understand exactly why the computer made a specific error.

Summary updated

BibTeX
@inproceedings{gokcen2018madly,
  author = {Ajda Gokcen and Ethan Hill and Michael White},
  title = {{Madly Ambiguous: A Game for Learning about Structural Ambiguity and Why It's Hard for Computers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-5011},
}
Figure