Choice-75: A Dataset on Decision Branching in Script Learning

Zhaoyi Joey Hou, Li Zhang, Chris Callison-Burch

TL;DR

Traditional script learning models narratives as rigid, linear sequences of events, ignoring the circumstantial choices humans make every day. This paper introduces Choice-75, a new benchmark dataset containing 75 scripts and over 600 scenarios designed to evaluate an AI's ability to handle decision branching. The evaluation reveals that while state-of-the-art LLMs excel at simple situational choices, they face a significant performance gap compared to humans on complex, multi-hop reasoning tasks.

What Problem Does This Paper Address?

Script learning is a subfield of AI focused on understanding how stereotypical events unfold—like the typical steps involved in ordering food at a restaurant or booking a flight. Historically, models have treated these scripts as linear chains of events. However, real life is full of crossroads where a script branches based on a person's unique circumstances.

For instance, if your goal is to "go to the grocery store," your next step might branch into "take a bus" or "call an Uber." The correct path depends entirely on the context: Are you trying to save money? Are you in a rush? Is it raining?

The authors address the lack of benchmarks for this problem by formalizing the task of decision branching. The goal is to challenge intelligent systems to look at a specific scenario and correctly identify the most reasonable path forward out of two options.

Methods

The authors frame decision branching as an in-context learning task. Given a goal, two options, and a situational scenario, a Large Language Model (LLM) must predict the correct choice: Option 1, Option 2, or Either (if the scenario makes the choice irrelevant).

To build a dataset with truly challenging scenarios, the authors used a human-in-the-loop generation paradigm with text-davinci-003 across two distinct formats:

  1. Verb Phrases via Multi-Hop Prompting: To simulate complex reasoning, they implemented a two-step prompting pipeline. The model first generates a simple base scenario leading to a choice (e.g., "wants to save money"), and then generates a deeper premise that causes that base scenario (e.g., "needs to buy a new car recently and doesn't have enough money").
Figure 2: Hard scenario generation (verb phrase)
  1. User Profiles: The model generates an unstructured, bulleted list of user traits (hobbies, financial status, occupation). This introduces realistic noise, forcing the system to parse a long context window and isolate the single piece of information relevant to the decision.
Figure 3: Hard scenario generation (user profile)

The models (text-davinci-003 and gpt-3.5-turbo) were evaluated using two prompt variations: a straightforward naive prompt and a narrative-driven story prompt.

Datasets

The authors introduced Choice-75, which uses the existing proScript dataset (a collection of 6.4k daily activity scripts) as a starting point. They sampled 75 goal actions and manually wrote two feasible options for each.

The final dataset consists of 620 scenarios annotated for difficulty based on the number of reasoning "hops" required to reach the answer:

  • Easy: Explicitly points to an option (1 hop).
  • Medium: Implicitly points to an option via basic commonsense (1-2 hops).
  • Hard: Requires multi-step reasoning and deep commonsense integration.
  • Either: Scenarios where both options are equally valid.

The scenario distribution across formats and difficulties is detailed below:

Distribution of scenarios across formats and difficulty levels in Choice-75.

Key Insights and Ideas

  • Human vs. Machine Difficulty Alignment: The authors' criteria for difficulty perfectly predicted LLM performance drops. As the number of reasoning hops increased, model accuracy degraded sharply, validating the dataset's design.
  • The "Either" Blindspot: LLMs struggle immensely when a scenario renders both choices equally acceptable. Instead of choosing "Either," the models display a strong bias toward arbitrarily picking one of the concrete options.
  • The Cost of Personalization: Moving from clean, descriptive verb phrases to unstructured user profiles caused a noticeable drop in LLM accuracy. This highlights the difficulty models face when filtering out irrelevant background noise to make personalized decisions.

Key Results

An example of the dataset's structure and the model evaluation can be seen in Figure 1:

An example of the Choice-75 dataset structure showing goals, options, and scenarios.
  • Overall Performance: The best-performing model setup (gpt-3.5-turbo with a story prompt) achieved an overall accuracy of 60%, falling significantly short of the 74% human baseline.
  • Easy vs. Hard Gap: While LLMs achieved up to 95% accuracy on easy manual verb phrases, their performance plummeted to 59%–68% on hard scenarios. Humans maintained a stable 76% accuracy on these same hard tasks.
  • The "Either" Failure: On scenarios evaluated as "Either," the best LLM scored a dismal 22%, compared to 53% for humans.

Limitations and Future Work

  • Dataset Scale and Domain: Choice-75 is relatively small and strictly confined to mundane, daily procedures inherited from proScript. Performance might vary wildly in specialized domains like medical or legal decision-making.
  • Oversimplified Binary Choices: The dataset assumes exactly two mutually exclusive options per goal. Real-world scripts often feature multiple overlapping, non-exclusive paths.
  • Prompt Engineering Constraints: The study only evaluated two basic prompt formats without exploring advanced techniques like Chain-of-Thought (CoT) prompting, which could potentially bridge some of the performance gaps in multi-hop reasoning.

Summary updated

Abstract

Script learning studies how stereotypical events unfold, enabling machines to reason about narratives with implicit information. Previous works mostly consider a script as a linear sequence of events while ignoring the potential branches that arise due to people's circumstantial choices. We hence propose Choice-75, the first benchmark that challenges intelligent systems to make decisions given descriptive scenarios, containing 75 scripts and more than 600 scenarios. We also present preliminary results with current large language models (LLM). Although they demonstrate overall decent performance, there is still notable headroom in hard scenarios.
BibTeX
@inproceedings{hou2024choice,
  author = {Zhaoyi Joey Hou and Li Zhang and Chris Callison-Burch},
  title = {{Choice-75: A Dataset on Decision Branching in Script Learning}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure