TL;DR
RAFiC is a new AI-driven framework that helps engineers build domain models by combining the broad knowledge of Large Language Models (LLMs) with the strict rules of Domain-Specific Languages (DSLs). By using an iterative "closed-loop" approach and a clever semantic mapping system, it provides suggestions that are not only syntactically correct but also contextually relevant, even for niche modeling languages the LLM hasn't seen much during training.
What Problem Does This Paper Address?
Domain modeling—the process of defining the concepts and relationships within a specific field (like aerospace or healthcare)—is a cornerstone of software engineering. However, it’s notoriously difficult, time-consuming, and prone to errors. While LLMs like GPT-4 or Llama have a wealth of general domain knowledge, they struggle with "low-resource" DSLs—languages that don't appear frequently in their training data.
The authors identify a "knowledge gap": an LLM might understand the concept of a hospital's ticketing system but won't know the specific syntax of a custom ticketing DSL or the intended modeling role of its elements. Furthermore, models are dynamic; a static prompt at the start of a session quickly becomes obsolete as the model evolves.
Methods
The authors introduce RAFiC (Recommender for AI-assisted Framework for domaIn model Completion). Its architecture rests on three pillars:
- Metamodel-Driven Guidance: It uses the DSL’s metamodel (the "grammar" of the language) to ensure that every suggestion the AI makes is structurally valid.
- Semantic-Role Annotation: This is the paper's most novel contribution. Instead of just giving the LLM class names, RAFiC maps DSL elements to a set of generic, high-level "modeling roles" (e.g., Process, Thing, Role, Goal). This acts as a translation layer, helping the LLM connect its general knowledge to the specific abstractions of the DSL.
- Iterative Orchestration: Rather than asking the LLM to "finish the model" in one go, RAFiC operates in a loop. It analyzes the current state of the model, identifies gaps, and generates targeted prompts for three types of actions:
- ARCH (Architectural): Introducing new concepts.
- REL (Relational): Connecting existing elements.
- CHAR (Characteristic): Adding attributes or constraints.
The system was implemented as a tool integrated into Sirius Web, a web-based modeling environment, allowing for real-time, interactive suggestions.
Datasets
The researchers evaluated RAFiC using three distinct DSLs, chosen to represent different levels of complexity and "familiarity" to standard LLMs:
- UML Class Diagrams (UML-CD): High-resource; very common in LLM training data. Scenarios: Medical Appointment and Food Delivery.
- Supply Chain Modeling (SCM): Medium-resource; process-oriented. Scenarios: Car Manufacturing and Aircraft Maintenance.
- SysML-v2: Low-resource; highly complex and strictly typed. Scenarios: Flashlight Assembly and BatMobile System Design.
The evaluation compared RAFiC against several baselines, including a "minimal" prompt and a "full" prompt where all language info is dumped into the LLM at once.
Key Insights and Ideas
- The "Right Info at the Right Time" Principle: The study found that iterative guidance—updating the prompt as the model grows—is far more effective than providing all the DSL knowledge upfront. This prevents "context drift," where the LLM loses track of the current modeling priorities.
- Semantic Roles as a Bridge: Mapping a specific class like "Ticket" to a generic role like "Goal" significantly improved the relevance of suggestions. It allows the LLM to reason: "If this is a Goal, I should probably suggest a Process to achieve it."
- The Familiarity Inverse: The less an LLM "knows" a language (like SysML-v2), the more it benefits from RAFiC’s guidance. For common languages like UML, the LLM’s internal knowledge is often "good enough," but for specialized industrial DSLs, explicit guidance is a game-changer.
Key Results
- Universal Improvement: Semantic role guidance improved the "contribution rate" (the percentage of the correct model recovered) in 100% of the tested scenarios.
- Massive Gains in Niche DSLs: In the Aircraft Maintenance scenario (SCM), the contribution rate jumped from 29% (minimal prompt) to 74% with RAFiC.
- Earlier Success: Cumulative acceptance curves showed that RAFiC helps modelers make progress much earlier in the session compared to baselines, which often stagnate after a few steps.
- Open-Source Viability: The results were achieved using Llama 3.1-70B, proving that well-guided open-source models can match or exceed the performance of larger commercial models in specialized engineering tasks.
Limitations and Future Work
- Redundancy: The authors noted that higher contribution rates often came with higher redundancy (the AI suggesting the same thing multiple times). Future versions need better "diversity control" to track what has already been explored.
- Manual Setup: While the semantic mapping is a one-time task for a language designer, it still requires manual effort. The authors suggest using LLMs to auto-generate these initial mappings in the future.
- Evaluation Scale: The study used two scenarios per DSL. While diverse, larger-scale user studies with human modelers are needed to measure actual productivity gains and "perceived usefulness" in real-world workflows.
Summary updated