MechBench: A Domain-Specific LLM Benchmark for Parallel Mechanisms

Zhihao Xia, Tingting Wang, Hui Li, Yongheng Xing, Chunxu TianarXiv 2026 · arXiv

TL;DR

MechBench is a new, rigorous benchmark of 2,541 questions designed to evaluate how well Large Language Models (LLMs) understand the complex physics and geometry of parallel mechanisms. The study reveals a startling "reasoning gap": while top-tier models can easily recite textbook knowledge, they struggle significantly when asked to perform executable engineering tasks like mechanism analysis and constrained design.

What Problem Does This Paper Address?

Parallel mechanisms—mechanical systems where an output platform is connected to a base by multiple independent kinematic chains—are notoriously difficult to model. Unlike simple serial robots, their behavior depends on a delicate interplay of topology, loop closure constraints, joint types, and specific geometries.

Existing LLM benchmarks like MMLU or GPQA are too broad to capture these nuances. They might test if a model knows the definition of "degrees of freedom," but they don't test if the model can actually design a four-limb mechanism that satisfies a specific motion screw requirement. The authors identify that current models often fail because they cannot maintain "constraint-consistency"—the ability to ensure that all parts of a complex mechanical system work together without violating physical laws.

Methods

The authors developed a comprehensive framework called MechForge to build the benchmark and a diagnostic model called MechGPT.

  • MechForge Workflow: This pipeline (see Figure 1) combines LLM-assisted knowledge organization with human-in-the-loop expert review. It uses a domain corpus of over 10,000 professional documents to generate questions that are then verified for physical validity.
  • Task-Aware Scoring: Instead of just checking if a model's text matches a reference, MechBench uses programmatic verification for its challenge tier. For design tasks, the model's output is treated as a "mechanism payload" and run through a checker to see if the proposed structure actually closes its loops and achieves the intended mobility.
  • MechGPT: To study how domain adaptation affects performance, the researchers fine-tuned a Qwen3-8B base model using continued pretraining (CPT) and supervised fine-tuning (SFT) on specialized mechanical engineering data.
Figure 1: MechForge workflow. Sources feed LLM-assisted knowledge organization and evidence-grounded or programmatic question generation. LLM checking and two-stage researcher review confirm conditions, answers, labels, and rubrics before tier assignment; reviewed QA can also form instruction data. Domain pretraining and QLoRA adapt MechGPT, which is evaluated by MechBench.

Datasets

The paper introduces MechBench, which is divided into three tiers:

  1. Standard Tier (1,791 questions): Focuses on professional knowledge across six categories (terms, topology, kinematics, modeling, diagnosis, and design trade-offs).
  2. Challenge Tier (550 questions): Covers nine executable tasks, including mobility analysis, joint replacement, and trajectory design (see Figure 3).
  3. Development Tier (200 questions): Used for protocol debugging.
Figure 3

The training data for MechGPT included a Domain Corpus of 10,379 documents and an Instruction Dataset of nearly 57,000 QA pairs.

Key Insights and Ideas

  • The "Book Smart vs. Street Smart" Gap: The most critical finding is that high performance on "knowledge" questions does not translate to "reasoning" capability. Models can be experts at explaining a theory while being unable to apply it to a specific, constrained problem.
  • Hard Gates in Design: Mechanical design has "hard gates"—if a model misses a single geometric constraint or joint axis direction, the entire mechanism fails. LLMs currently struggle with this level of precision.
  • Programmatic Verification is the New Gold Standard: The authors argue that for engineering domains, simple rubrics aren't enough. We need executable checks to see if an AI’s "design" actually works in a simulated environment.
  • Domain Adaptation is a Double-Edged Sword: While fine-tuning (MechGPT) significantly improved performance on standard knowledge over the base model, it still left a massive gap in the challenge tier compared to massive closed-source models.

Key Results

The quantitative results highlight the difficulty of the domain:

  • Standard vs. Challenge: Leading models like GPT-5.6-Sol achieved scores over 95% on the standard tier but plummeted to between 47% and 60% on the challenge tier.
  • MechGPT Performance: The domain-adapted MechGPT (8B) scored 62.35 on the standard tier (beating its base model's 42.94) but only 18.30 on the challenge tier.
  • Task Failure: In the "Specified trajectory design" task, no evaluated model obtained any credit, showing that jointly synthesizing topology, closure, and trajectory is currently beyond LLM capabilities (see Figure 6).
  • Constraint Sensitivity: Even the best models struggled with "output screw space" checks, succeeding only ~24% of the time, proving that constructing a geometry-consistent motion basis is a major hurdle.
Figure 6

Limitations and Future Work

The authors acknowledge several limitations:

  • Text-Only Input: MechBench currently relies on text and structured representations. It cannot yet "see" CAD drawings or hand-sketched mechanism diagrams, which are central to real-world engineering.
  • Corpus Quality: The framework doesn't yet automatically filter the quality of the pretraining corpus, still requiring significant expert oversight.
  • Reasoning Depth: Domain adaptation helps with facts, but deeper structural reasoning remains a challenge.

Future directions include moving toward multimodal inputs (interpreting drawings), integrating geometric representations directly into training, and adopting even stronger open-weight foundation models.

Summary updated

Abstract

Parallel mechanisms pose distinctive challenges to large language models (LLMs), as valid reasoning depends jointly on mechanism topology, loop closure, geometry, joint types, and modeling assumptions. Yet existing LLM benchmarks provide little systematic coverage of such domain-specific knowledge, constraint-sensitive analysis, and constructive design. To fill this gap, we introduce MECHBENCH, a 2,541-question benchmark for evaluating LLMs in parallel mechanisms across professional knowledge, mechanism analysis, and design. MECHBENCH contains standard and challenge tiers spanning six capability categories and nine executable analysis and design tasks. It is constructed through MECHFORGE, a domain-grounded framework that combines expert-organized knowledge, LLM-assisted question generation, programmatic instance generation, two-stage expert review, and task-aware scoring with exact matching, rubric-based evaluation, and program verification. We further develop MECHGPT as a diagnostic domain-adapted model. Experiments show that strong knowledge performance does not directly translate into executable mechanism reasoning: leading closed-service models achieve over 95 on the standard tier but only 47-60 on the challenge tier, while domain adaptation substantially improves MechGPT over its Qwen3-8B baselines but leaves a large gap in constrained analysis and design. These results establish MECHBENCH as a systematic testbed for measuring domain-specific LLM capabilities in mechanism science.
BibTeX
@misc{xia2026mechbench,
  author = {Zhihao Xia and Tingting Wang and Hui Li and Yongheng Xing and Chunxu Tian},
  title = {{MechBench: A Domain-Specific LLM Benchmark for Parallel Mechanisms}},
  year = {2026},
  eprint = {2611.15342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15342},
}
Figure