An Experimental Framework for Designing Document Structure for Users' Decision Making: An Empirical Study of Recipes

Rina Kagawa, Masaki Matsubara, Rei Miyata, Takuya Matsuzaki, Yukino Baba, Yoko Yamakata

TL;DR

This paper introduces a novel experimental framework to design document structures that actually help laypeople make decisions, using cooking recipes as a primary case study. By crowdsourcing insights from over 1,300 amateur cooks, the researchers identified that specific types of information—like ingredient tips and procedural notes—significantly boost a document's perceived usefulness when placed at the end of specific steps and highlighted with subheadings.

What Problem Does This Paper Address?

We live in an era of asynchronous communication where documents (manuals, textbooks, recipes) are meant to guide our actions. However, there is a persistent "expert-layperson gap": experts who write these documents often fail to include the specific information that beginners need to make flexible decisions.

Traditional metrics for document quality usually focus on readability (how easy is it to read?) or accuracy (did the reader understand the facts?). But for a document to be truly effective for decision-making—like deciding to swap salmon for tuna in a recipe—it needs to be useful. The authors argue that we lack a systematic way to identify what content laypeople find useful and how that content should be arranged to maximize its impact.

Methods

The researchers developed a two-phase experimental framework:

  1. Content Identification (The Three-Step Procedure):
    • Collection: 20 lay readers compared pairs of recipes and described in their own words what made one more useful than the other, generating 426 "content candidates."
    • Aggregation: 370 participants grouped these candidates by similarity. The researchers then used the Louvain method—an unsupervised community detection algorithm—to cluster these into distinct categories without pre-defining them.
  2. Conceptualization: 50 participants voted on names for these clusters, resulting in six core content categories.
The three-step experimental procedure for extracting categories of useful content.
  1. Structural Evaluation: The team created "deleted" and "rearranged" versions of recipes to test 36 different structures (6 content types × 6 arrangements). They used multi-level modeling (fitted via Markov chain Monte Carlo chains) to analyze how these structures affected "usefulness scores," while accounting for variables like the reader's background knowledge and the presence of unknown words.
Our proposed framework to define a preferred document structure and research questions.

Datasets

The study utilized the Rakuten Dataset, a massive collection of Japanese online recipes.

  • Professional Recipes: 9,820 recipes written by professional chefs.
  • Amateur Recipes: 113,688 recipes submitted by users.
  • Participants: 1,340 amateur cooks recruited via the Japanese crowdsourcing platform Lancers.

Key Insights and Ideas

  • The Power of "Why" and "What If": Laypeople don't just want steps; they want context. The identified categories—Reasons, Suitable Ingredients, Notes, Substitutions, Variations, and Servings—all provide the "meta-information" necessary for a reader to adapt a document to their specific situation.
  • Arrangement is as Important as Content: Simply including a helpful note isn't enough. The study found that the placement of this information significantly alters its perceived value.
  • Subheadings as Cognitive Signposts: Using explicit subheadings (e.g., <Notes>) at the end of a specific cooking step was the most effective way to increase usefulness. This likely reduces cognitive load by signaling exactly when and where the supplemental information is relevant.

Key Results

The framework successfully distilled six categories of useful content. The quantitative analysis revealed:

  • Suitable Ingredients and Notes had the most significant impact on usefulness.
  • Arrangement A01 (placing the content at the end of each cooking step with a clear subheading) was the "gold standard." For example, adding notes in this format increased usefulness scores with a mean effect of 0.03 (95% CI [0.01, 0.04]).
  • Interestingly, while "Servings" and "Reasons" were identified as useful categories, their specific arrangement didn't show the same statistically significant boost as ingredients and notes, suggesting some information types are more sensitive to layout than others.

Limitations and Future Work

The authors acknowledge several constraints:

  • Reading vs. Doing: The study measured perceived usefulness while reading, not whether the readers actually cooked better meals. Future work should involve task-based evaluations in real-world settings.
  • Linguistic and Cultural Bias: The study was conducted entirely in Japanese. Document structure preferences may vary across different cultures and languages.
  • Stylistic Nuance: The research focused on what and where, but not how. The tone, vocabulary, and specific phrasing of the sentences were not analyzed, which likely also play a role in a document's effectiveness.

Summary updated

Abstract

Textual documents need to be of good quality to ensure effective asynchronous communication in remote areas, especially during the COVID-19 pandemic. However, defining a preferred document structure (content and arrangement) for improving lay readers' decision-making is challenging. First, the types of useful content for various readers cannot be determined simply by gathering expert knowledge. Second, methodologies to evaluate the document's usefulness from the user's perspective have not been established. This study proposed the experimental framework to identify useful contents of documents by aggregating lay readers' insights. This study used 200 online recipes as research subjects and recruited 1,340 amateur cooks as lay readers. The proposed framework identified six useful contents of recipes. Multi-level modeling then showed that among the six identified contents, suitable ingredients or notes arranged with a subheading at the end of each cooking step significantly increased recipes' usefulness. Our framework contributes to the communication design via documents.
BibTeX
@inproceedings{kagawa2024experimental,
  author = {Rina Kagawa and Masaki Matsubara and Rei Miyata and Takuya Matsuzaki and Yukino Baba and Yoko Yamakata},
  title = {{An Experimental Framework for Designing Document Structure for Users' Decision Making: An Empirical Study of Recipes}},
  booktitle = {ICADL},
  year = {2024},
  doi = {10.1007/978-981-96-0865-2\_6},
}
Figure