Evaluating LLM-Generated Diagrams as Graphs

Chumeng Liang, Jiaxuan YouEMNLP 2025

TL;DR

This paper introduces DiagramEval, a novel graph-based evaluation metric for assessing the quality of diagrams generated by Large Language Models (LLMs). By treating diagrams as graphs and analyzing node and path alignments, DiagramEval offers a more fine-grained and interpretable evaluation than existing methods, revealing shortcomings in LLM-generated diagrams that were previously overlooked.

What Problem Does This Paper Address?

Diagrams are crucial for conveying complex ideas in research papers, but their creation is labor-intensive. While LLMs show promise for automated diagram generation, evaluating the quality of these generated diagrams is challenging. Existing metrics, often borrowed from general text-to-image tasks, are coarse-grained, lack explainability, and can be overly sensitive to visual elements like spatial layout rather than the logical correctness of the diagram's content. This makes it difficult to truly understand the strengths and weaknesses of LLM-generated diagrams and guide further improvements.

Methods

The core of DiagramEval is to represent diagrams as graphs. This involves two main steps:

  1. Constructing Diagram Graphs:
    • Node Extraction: Text elements within SVG diagram files are identified as nodes. This process combines spatial and semantic analysis, leveraging a multi-modal LLM to refine the initial extraction by merging, adding, or removing nodes based on their textual meaning and visual context.
  2. Edge Extraction: Connections (like arrows and lines) between nodes are identified. This is achieved by using an LLM with vision capabilities, which analyzes visual cues in the diagram image and the identified nodes to determine directed edges.
  3. Evaluation Metrics:
    • Node Alignment: This metric assesses the fidelity of textual content by comparing the set of text elements (nodes) in the generated diagram with those in the ground truth. It uses precision, recall, and F1 scores based on a node-matching procedure that considers textual similarity.
  4. Path Alignment: This metric evaluates the structural fidelity by examining the consistency of paths (sequences of directed edges) between matched nodes in both the generated and ground truth diagrams. It also uses precision, recall, and F1 scores, focusing on whether relationships between elements are preserved.

Datasets

The authors collected a dataset of 361 diagrams from papers accepted to CVPR2025. These papers were chosen because they were released after the data cutoff date of the LLMs used, preventing knowledge leakage. An automated pipeline was used to select diagrams with abundant text annotations and collect them along with their captions and corresponding paper context.

Key Insights and Ideas

  • Diagrams as Graphs: Conceptualizing diagrams as graphs (nodes for text elements, edges for connections) provides a powerful framework for detailed structural analysis.
  • LLMs Struggle with Logical Flow: Even state-of-the-art LLMs can generate diagrams that look plausible but have incorrect or missing logical connections, which DiagramEval can detect.
  • Limitations of CLIPScore: Existing metrics like CLIPScore are sensitive to superficial visual elements (layout, icons) and can be "hacked" by LLMs that generate diagrams that match captions but not the underlying logic. DiagramEval offers a more robust and interpretable alternative.
  • Explainability is Key: DiagramEval's fine-grained metrics provide specific insights into why a diagram is good or bad, highlighting issues with node alignment (textual content) or path alignment (logical flow).

Key Results

  • DiagramEval's Effectiveness: The proposed metrics (Node Alignment and Path Alignment) were used to evaluate three LLMs (Llama 4 Maverick, Gemini 2.5 Pro, and Claude 3.7 Sonnet). The results showed that Claude 3.7 Sonnet performed best across most metrics.
  • Correlation with CLIPScore: Node Alignment metrics showed a positive correlation with CLIPScore, indicating they capture similar aspects of textual consistency. However, DiagramEval's metrics provided healthier score distributions and better explainability. Path Alignment metrics showed minimal correlation with CLIPScore, highlighting its ability to capture structural logic that CLIPScore misses.
  • Human Evaluation Alignment: DiagramEval metrics showed better alignment with human judgments compared to CLIPScore, suggesting they are more trustworthy for evaluating the semantic similarity and logical correctness of diagrams.
  • Case Studies: The paper presents cases where DiagramEval and CLIPScore differ, demonstrating how DiagramEval can identify issues like incorrect logical flow (low Path F1) or sensitivity to irrelevant visual elements (low CLIPScore Text).

Limitations and Future Work

  • LLM Performance Uncertainty: The accuracy of DiagramEval is partly dependent on the LLM's ability to reliably extract edges from reference graphs. If the LLM misses edges in the ground truth, it can lead to underestimated evaluations.
  • Future Work: Addressing the limitations in LLM performance for edge extraction is a key direction for future work. The authors also suggest exploring how to further reduce the dependency on complex models and enhance interpretability.

Summary updated

BibTeX
@inproceedings{liang2025evaluating,
  author = {Chumeng Liang and Jiaxuan You},
  title = {{Evaluating LLM-Generated Diagrams as Graphs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}

Loading related content...

Figure