TL;DR
This paper introduces MCeT, a novel automated tool that uses Large Language Models (LLMs) to evaluate the correctness of behavioral models (specifically sequence diagrams) against textual requirements. By breaking down the evaluation into fine-grained, multi-perspective checks and employing a self-consistency mechanism, MCeT significantly improves the detection of issues compared to direct LLM comparisons, offering more precise and comprehensive feedback for model-driven engineering.
What Problem Does This Paper Address?
In model-driven engineering, behavioral models like sequence diagrams are crucial for documenting system dynamics. However, ensuring these models accurately reflect their corresponding textual requirements is a significant challenge. Manually verifying these models is time-consuming and error-prone. While LLMs are emerging as powerful assistants for generating these models, their use in automatically evaluating model correctness against natural language requirements is still nascent. Existing automated methods often rely on formal semantics or reference models, which are not always available or may not account for valid alternative models. This paper addresses the gap in automated, natural language-based evaluation of behavioral models against free-style requirements, aiming to provide engineers and AI assistants with reliable feedback.
Methods
MCeT employs a two-stage approach to evaluate sequence diagrams against textual requirements:
- MCeT-A (Atomic Checking): This stage breaks down the evaluation into three complementary checks:
- Holistic Check: The entire diagram is compared against the entire requirements text using an LLM.
- Diagram-atom Check: The diagram is decomposed into "diagram-atoms" (message + participants), and each atom is checked for accuracy and completeness against the requirements, considering its context within the diagram.
- Requirement-atom Check: Requirements are split into "requirement-atoms" (atomic requirements), and the diagram is checked for complete and accurate implementation of each.
- A voting mechanism is used within each check to aggregate multiple LLM responses and improve reliability.
- MCeT-X (Higher Authority Cross-check): This stage mitigates LLM hallucinations by cross-checking the results from MCeT-A. It leverages the observation that requirement-atom checks tend to have higher precision. Issues flagged by diagram-atom or holistic checks are cross-checked against "correct requirement-atoms" (those with no issues reported). If a diagram-atom or holistic issue conflicts with a requirement-atom deemed correct, it's considered a hallucination and discarded. This approach combines the strengths of different checks to improve overall accuracy.
Datasets
The evaluation was performed on the FBENCH dataset, which consists of 76 requirement-diagram pairs derived from real-world industrial projects. These pairs were augmented with requirement "smells" (ambiguity, inconsistency, incompleteness) that introduce issues into the generated diagrams. The dataset includes issues reported by experienced engineers, serving as ground truth for evaluating MCeT's performance.
Key Insights and Ideas
- Holistic LLM evaluation is insufficient: Directly asking an LLM to compare an entire diagram to requirements misses a significant portion of issues that experienced engineers can identify.
- Decomposition is key: Breaking down both the diagram and requirements into atomic units (diagram-atoms and requirement-atoms) allows LLMs to focus on specific interactions and details, leading to more accurate evaluations.
- Requirement-atom checks are high-authority: The requirement-atom check, which verifies if the diagram implements atomic requirements, demonstrates higher precision and recall, making it a valuable component for cross-checking.
- Self-consistency mitigates hallucinations: The cross-checking mechanism in MCeT-X effectively filters out false positives by leveraging the higher reliability of requirement-atom checks, significantly improving precision.
- LLMs can provide detailed, natural language feedback: MCeT not only detects issues but also provides explanations in natural language, making the feedback actionable for engineers.
Key Results
- MCeT achieves a precision of 0.81, a significant improvement over the baseline holistic approach's precision of 0.58.
- MCeT detects 90% more issues that experienced engineers found compared to the direct holistic approach.
- The tool reports an average of 6 new issues per diagram that were not identified by human experts.
- In terms of recall, MCeT's combined results (MCeT-A) achieve 68.1% recall, nearly doubling the baseline holistic check's 34.1% recall.
- MCeT identifies 391 new issues not reported by human experts, a 14x increase over the baseline.
- The cross-checking stage (MCeT-X) further refines the results, achieving a precision of 0.81 and reducing false positives by 428 to 217.
- The approach is robust across different LLMs, with lighter-weight models like GPT-40-mini achieving comparable performance to more powerful models at a fraction of the cost.
Limitations and Future Work
- LLM Probabilistic Nature: LLMs are probabilistic, meaning repeated runs might yield slightly different results. The paper addresses this with a voting mechanism.
- Generalization to Other Model Types: The current study focuses on sequence diagrams. Future work aims to generalize the approach to other behavioral model types.
- Preventing Over-filtering: The authors plan to investigate preventing the cross-checking approach from incorrectly discarding real issues.
- Subjectivity in Evaluation: The evaluation relies on human judgment, which can be subjective. The authors mitigated this by using multiple annotators and calculating inter-rater reliability.
- Dataset Size for LLM Trade-offs: The LLM trade-off analysis was conducted on a subset (10%) of the dataset due to computational costs.
Summary updated