Do AI Assistants Help Students Write Formal Specifications? A Study with ChatGPT and the B-Method

Alfredo Capozucca, Daniil Yampolskyi, Alexander Goldberg, Maximiliano Cristiá

Introduction

Formal Methods (FM) are mathematically based techniques essential for the specification, development, and verification of safety-critical software and hardware systems. Despite their importance, the B-Method—a state-based formal language—is notoriously difficult for students to master due to its requirement for extreme mathematical precision. With the rise of Large Language Models (LLMs), educators are increasingly questioning whether AI assistants (AIAs) like ChatGPT serve as a helpful bridge for learning these complex languages or as a "crutch" that undermines logical rigor.

This study investigates the impact of ChatGPT on undergraduate students' ability to write formal B-specifications, comparing performance across manual and AI-assisted workflows.

Methodology

The researchers conducted a two-year, single-group pretest-posttest quasi-experiment at the University of Luxembourg. The participants were undergraduate computer science students with no prior experience in the B-Method.

Experimental Setup

  • Tasks: Students translated English requirements into B-specifications using the Atelier B tool.
  • Treatment: In the pretest phase, students worked without AI. In the posttest phase, they were encouraged to use ChatGPT (GPT-3.5 in 2023; GPT-4 in 2024).
  • Evaluation: Submissions were graded across six dimensions: state variables, initialization, abstraction level, operation specification, invariants, and type checking.
  • Trust and Adoption: The team measured "Normalised Edit Distance" (NED) to determine how much of the AI's output was actually used in the final submission, comparing this against self-reported trust levels.

Key Findings

The Performance Gap

Surprisingly, the introduction of AI did not lead to better outcomes. In both iterations of the study, average correctness scores dropped when students used the AI. In the first iteration (GPT-3.5), scores fell from a 77.9% baseline to 59.8%. While GPT-4 (Iteration 2) performed better than its predecessor, the median performance with AI assistance still did not exceed the "no-AI" baseline.

The Trust Paradox

A primary insight from the study is the negative correlation between trust and performance. Students who expressed low trust in the AI and used its output sparingly (resulting in a high NED) produced significantly better specifications. Those who "AI-adopted" more heavily—copying logic without rigorous verification—suffered the most significant performance declines.

Brainstorming vs. Logical Execution

ChatGPT demonstrated a clear split in utility:

  • Strengths: The AI was relatively effective during the "discovery" phase, helping students identify necessary state variables and operations from natural language descriptions.
  • Weaknesses: The AI struggled significantly with the "writing" phase. It consistently failed to generate syntactically and logically correct B-code, particularly regarding invariants. Since invariants represent the logical properties that must always hold true, the AI's failure here suggests a lack of deep understanding of the system's state space.

Behavioral Patterns in Prompting

The researchers identified a specific prompting sequence that correlated with more successful outcomes. This "successful" pattern involved:

  1. Warmup: Providing initial context.
  2. Language: Explicitly specifying the target formal language.
  3. Helper: Providing examples to guide the AI.
  4. Instructions/Commands: Iteratively refining the output with specific logical constraints.

Conclusion and Future Work

The study concludes that while LLMs are useful brainstorming partners for identifying domain concepts, they are currently unable to handle the rigorous logical demands of formal methods. For students, over-reliance on AI leads to a decay in specification quality.

Future research will focus on developing specific "prompt engineering" curricula for Formal Methods, teaching students how to interleave human logic with AI-generated suggestions. For now, the authors advise that the "heavy lifting" of formal logic remains a human necessity, and educators need not fear that AI will make formal methods assessments obsolete in their current form.

Summary updated

BibTeX
@inproceedings{capozucca2025do,
  author = {Alfredo Capozucca and Daniil Yampolskyi and Alexander Goldberg and Maximiliano Cristiá},
  title = {{Do AI Assistants Help Students Write Formal Specifications? A Study with ChatGPT and the B-Method}},
  booktitle = {CSEE\&T},
  year = {2025},
  doi = {10.1109/cseet66350.2025.00009},
}
Figure