NEBench R1: a Reasoning Benchmark for Evaluating Nuclear Domain Competence in Large Language Models

Liyan Wang, Longze Li, Mohamed Hassan, Isshu Lee, Dajie Sun, Ruimin Feng, Aleksandar VakanskiarXiv 2026 · arXiv

TL;DR

NEBench R1 is a new, large-scale benchmark designed to test whether Large Language Models (LLMs) actually understand nuclear engineering or are just lucky guessers. By evaluating models on over 7,500 U.S. Nuclear Regulatory Commission (NRC) exam questions, researchers found that even the best models fail to meet passing standards and that longer "thinking" times often signal a model is about to get the answer wrong.

What Problem Does This Paper Address?

As LLMs are increasingly proposed for safety-critical roles—like assisting reactor operators or reviewing technical specifications—the stakes for "hallucinations" become incredibly high. In nuclear engineering, a correct answer is useless if it’s based on flawed logic.

Existing benchmarks in this field are often too small or focus solely on multiple-choice accuracy. A model might pick the right letter through memorization or process of elimination without understanding the underlying thermodynamics or reactor kinetics. The authors address this gap by introducing NEBench R1, which pairs thousands of questions with official technical explanations to evaluate the quality of a model's reasoning.

Methods

The researchers evaluated six prominent LLMs (including Llama 3.3, Qwen 3.8, Grok 4.6, and Claude Opus 5.5) using a two-pronged approach:

  1. Zero-Shot Accuracy: Standard multiple-choice testing across the entire dataset.
  2. Rubric-Based Reasoning Evaluation: For questions answered correctly, an "automated judge" (GPT-5.6-Terra) used a five-dimensional rubric to grade the model's explanation. The dimensions included:
  3. Problem Comprehension (D2a): Did it understand the scenario?
  4. Theory/Operational Understanding (D2b): Did it apply the right physics?
  5. Factual/Procedural Accuracy (D2c): Were the specific numbers and steps correct?
  6. Distractor Analysis (D3): Could it explain why the wrong answers were wrong?
  7. Clarity and Technical Precision (D4/D5): Was the response well-organized and terminologically sound?

For "reasoning" models like Grok and Claude, the team also analyzed internal "thinking" tokens to see how inference-time effort correlated with success.

Datasets

The study introduces NEBench R1, a benchmark of 7,546 unique multiple-choice questions extracted from official NRC reactor operator licensing examinations.

  • Coverage: 18 different nuclear facilities, including both Boiling Water Reactors (BWRs) and Pressurized Water Reactors (PWRs).
  • Classification: Questions are mapped to the NRC’s "Knowledge and Abilities" (K/A) catalogs, covering areas like Plant Systems, Emergency Evolutions, and Thermodynamics.
  • Uniqueness: Unlike previous datasets, every question includes an authoritative explanation used as a ground-truth reference for reasoning.
Fig. 1. Example of a question, correct answer, and explanation.

Key Insights and Ideas

  • The "Thinking" Paradox: In frontier models like Grok 4.6 and Claude Opus 5.5, more "thinking" tokens actually correlated with lower accuracy. This suggests that reasoning length is a proxy for question difficulty; if a model has to "think" a lot, it’s likely struggling with a concept it doesn't fully grasp.
  • Fluency is Deceptive: Models often scored high on "Clarity and Relevance" (D4) while scoring poorly on "Factual Accuracy" (D2c). This is a major red flag for safety-critical fields: a model can sound incredibly confident and professional while being technically wrong.
  • Distractor Analysis is the Ultimate Test: Even top-tier models struggled to explain why incorrect options were wrong (D3). This indicates that their "knowledge" may be more about pattern matching the correct path than truly understanding the boundaries of the system.
  • Position Bias: The researchers found a striking bias in NVIDIA’s Nemotron-Ultra-550B, which chose option "D" for over 42% of all questions, regardless of the actual correct answer.
Fig. 2. Predicted answer-choice distributions for each model

Key Results

  • Performance Tiers: The models fell into three distinct tiers. Llama 3.3 and Qwen 3.8 sat at the bottom (~34% accuracy), while Claude Opus 5.5 led the pack at 73.9%.
  • The Passing Gap: Even the best model (73.9%) failed to reach the typical 80% NRC passing threshold required for human operators.
  • Reasoning vs. Accuracy: For Grok 4.6, accuracy plummeted from 90.3% (for responses under 500 reasoning tokens) to just 39.0% (for those exceeding 20,000 tokens).
Grok 4.6 shows a clear decline in accuracy as the number of reasoning tokens increases.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Multiple Choice Limits: The format allows for lucky guesses (25% baseline), which doesn't fully capture open-ended engineering reasoning.
  • Data Contamination: Since NRC exams are public, there is a risk that these questions were part of the models' original training data.
  • Judge Bias: Using one LLM to grade another can introduce systematic biases.

Future work will likely focus on multimodal understanding (interpreting reactor diagrams and charts) and moving toward open-ended procedural generation to better simulate real-world nuclear engineering tasks.

Summary updated

Abstract

Large language models (LLMs) are increasingly being explored for knowledge-intensive and safety-critical applications in nuclear engineering. However, existing nuclear-domain benchmarks provide limited content coverage or focus primarily on answer accuracy without systematically evaluating the quality of the underlying reasoning. This study introduces NEBench R1, a reasoning-oriented benchmark comprising 7,546 unique multiple-choice questions extracted from U.S. Nuclear Regulatory Commission (NRC) reactor operator licensing examinations covering 18 nuclear facilities. Each question is classified according to the NRC Knowledge and Abilities catalogs and accompanied by an official explanation serving as a reference for reasoning evaluation. Six LLMs were evaluated, with overall accuracy ranging from 34.0% to 73.9% across three distinct performance tiers. To assess reasoning quality, an LLM-based judge evaluated correctly answered responses using a multidimensional rubric. Although responses generally demonstrated strong problem comprehension, they scored lower in factual and procedural accuracy and distractor analysis, indicating that correct answers were not necessarily supported by technically sound justifications. Response clarity was also weakly indicative of technical quality. Furthermore, longer reasoning traces were associated with lower accuracy in both frontier reasoning models, suggesting that reasoning length may reflect question difficulty rather than improved problem-solving performance. Reasoning-token consumption may therefore provide an additional signal for identifying potentially unreliable responses. NEBench R1 establishes a benchmark for evaluating both answer accuracy and reasoning quality, providing a foundation for assessing the domain competence and reliability of LLMs in nuclear engineering.
BibTeX
@misc{wang2026nebench,
  author = {Liyan Wang and Longze Li and Mohamed Hassan and Isshu Lee and Dajie Sun and Ruimin Feng and Aleksandar Vakanski},
  title = {{NEBench R1: a Reasoning Benchmark for Evaluating Nuclear Domain Competence in Large Language Models}},
  year = {2026},
  eprint = {2611.21682},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21682},
}
Figure