TL;DR
This paper introduces a novel method for generating faithful educational diagrams from text by using Rhetorical Structure Theory (RST) to guide In-Context Learning (ICL). By selecting prompt examples based on underlying discourse structure rather than surface-level keywords, the approach significantly reduces AI hallucinations and context inconsistencies. The study demonstrates that while standard LLMs struggle with these complex structural mappings, advanced reasoning models like OpenAI's o3 excel, producing highly accurate and reliable visualizations.
What Problem Does This Paper Address?
Generative AI is increasingly being integrated into online learning platforms to automatically turn textbook text into visual aids like flowcharts and concept trees. However, large language models (LLMs) frequently suffer from "faithfulness hallucinations"—they generate diagrams that contradict the source text, omit crucial steps, or fabricate entirely new information. In an educational setting, these errors pose a significant risk to students.
The core challenge lies in mapping the abstract, hierarchical organization of a text to a syntactically correct and visually logical diagram code (such as Graphviz DOT syntax). Current automated systems struggle to infer the correct diagram type (e.g., a flowchart for sequential steps vs. a tree diagram for hierarchical lists) purely from raw text without human intervention.
Methods
The authors address this by leveraging Rhetorical Structure Theory (RST), a framework from computational linguistics that models text coherence through hierarchical relationships between elementary discourse units (EDUs).
The authors designed a structured diagram generation pipeline consisting of five main steps:
- RST Analysis: The LLM analyzes the source text to extract its discourse structure and relations.
- Similarity Search: The system searches a pre-analyzed pool of examples to find a text with a matching rhetorical structure (e.g., identifying texts that both rely heavily on sequential or list-based relations).
- Example Construction: The system builds an In-Context Learning (ICL) prompt using two distinct strategies:
- RST1: Pairs a standard text example with its corresponding diagram code.
- RST2: Pairs an abstract RST analysis directly with the diagram code. This forces the model to perform an Out-of-Distribution (OOD) task, learning a direct mapping from discourse structure to code.
- ICL Generation & Self-Repair: The model generates DOT code. If the code contains rendering bugs, an automated loop attempts to repair it (up to 5 times).
- Diagram Refinement: A final prompt instructs the model to optimize layout aesthetics (e.g., minimizing line crossings).
The complete architecture of these pipelines is visualized below in Figure 1:
Datasets
- The Contributed Dataset: The authors introduced a newly curated dataset containing 150 text-diagram pairs. These were generated from 25 distinct educational source texts spanning computer science and programming topics.
- Evaluation Setup: Each of the 25 texts was processed through the three pipeline variations (0-shot, RST1, RST2) across two state-of-the-art models (
gpt-4o-2025-03-26ando3-2025-04-03), creating a robust benchmark for expert and automated evaluation.
Key Insights and Ideas
- Structure Trumps Semantics: Matching in-context examples by their rhetorical structure rather than semantic content naturally guides the LLM toward the correct visualization format. For instance, a text dominated by sequential relations correctly triggers a flowchart, while list relations trigger a tree diagram.
- Suppressed Priors via OOD Tasks: Forcing the model to generate code directly from an abstract discourse analysis (the RST2 method) acts as a powerful constraint. It prevents the model's pre-trained internal knowledge (its "prior") from dominating the output, effectively shutting down its ability to introduce outside, fabricated facts.
- Reasoning Capabilities are Mandatory for Complex ICL: Standard models like
gpt-4ofrequently fail when given abstract structural prompts (RST2), leading to logical contradictions. Conversely, reasoning-optimized models likeo3thrive on these out-of-distribution tasks, proving that advanced reasoning is a prerequisite for reliable structural text-to-diagram translation.
Key Results
The generated diagrams were evaluated by human experts across three criteria: Logical Organization ($C_1$), Connectivity ($C_2$), and Layout Aesthetic ($C_3$). The results were analyzed using Bayesian Generalized Linear Mixed Models (GLMMs):
- Reduction in Factual Fabrications: The RST2 method drastically reduced factual hallucinations, yielding an Odds Ratio of 0.16 compared to the baseline.
- Improved Faithfulness: Both RST1 and RST2 significantly lowered text inconsistency rates. However, as shown in Figure 2, the severity of faithfulness hallucinations depended heavily on the model, with
o3showing a clear advantage.
- Clear Superiority of
o3: Theo3model substantially outperformedgpt-4oacross all metrics. Figure 3 highlights the calculated odds ratios, showing thato3significantly increases the probability of generating a high-quality diagram ($G$).
- Automated Evaluation Success: The authors tested automated grading pipelines using LLMs. When provided with explicit rubric instructions,
o3achieved statistically significant agreement with human expert ratings, offering a viable path toward automated quality control.
Limitations and Future Work
- Sample Size: The study relied on a relatively small sample of 25 source texts, which introduced wide credible intervals in some of the statistical estimates.
- Incomplete Rhetorical Coverage: The small pool of in-context examples did not cover the entire taxonomy of possible rhetorical relations found in natural language.
- Layout Refinement Bottleneck: The authors noted that LLMs are still highly ineffective at refining layout aesthetics (like fixing line bends or overlapping elements) through text prompts alone, occasionally introducing new visual bugs during the refinement phase.
- Future Directions: The authors recommend expanding the dataset, exploring mechanistic interpretations of how ICL processes structural prompts, and adapting this method to other multimodal generation tasks.
Summary updated