CivilBench: Benchmarking Vision-Language Models for Multimodal Information Integration in Civil Engineering

Dan Yuan, Yue Wen, Zhili He, Zhenkui Zhou, Shicen Liu, Ziqiang ZhengarXiv 2026 · arXiv

While Vision-Language Models (VLMs) like GPT-4 and Gemini have mastered general-purpose tasks, their "professional" utility in specialized fields remains a major question mark. In a new paper, researchers introduce CivilBench, the first comprehensive benchmark designed to test whether these models can actually handle the rigors of civil and structural engineering.

TL;DR

CivilBench is a new 1,000-sample benchmark that evaluates VLMs across three critical engineering domains: sequential reasoning, structural understanding, and open-world decision support. The results show a massive "reasoning gap"—while models are decent at identifying visible damage, they struggle significantly with the numerical and ordinal reasoning required for professional engineering judgments.

What Problem Does This Paper Address?

Civil engineering is a "knowledge-intensive" and "safety-critical" domain. Unlike identifying a cat in a photo, an engineering assistant must interpret complex visual cues (like crack patterns), understand physical mechanisms (how loads transfer), and make quantitative risk assessments.

Existing multimodal benchmarks are too general. They might test if a model can see a bridge, but they don't test if the model can determine if that bridge is at risk of imminent collapse based on a sequence of inspection reports. The authors identify a lack of rigorous, domain-specific evaluation tools to measure if VLMs are ready for real-world infrastructure monitoring and disaster response.

Methods

The authors developed a systematic pipeline to ensure CivilBench is both challenging and "visually necessary."

  1. Task Taxonomy: They organized the benchmark into three "families" (see Figure 1):
  2. Sequential Reasoning: Inferring future states or ranking risks from sparse, structured historical data.
  3. Image-based Structural Understanding: Identifying materials, counting defects, and localizing damage.
  4. Open-world Understanding: High-level tasks like hazard inference, causal analysis, and repair planning.
  5. The Visual Necessity Test: To prevent models from "cheating" using linguistic shortcuts (answering based on text alone), every sample underwent a test where the image was removed. If a model could answer correctly without the image, the sample was discarded (see Figure 2).
  6. Expert Annotation: Every question was reviewed by structural engineering experts to ensure technical accuracy and practical relevance.
  7. Evaluation: The researchers tested 17 representative VLMs, including heavyweights like GPT-4o, Gemini 1.5 Pro, and open-source contenders like Qwen2.5-VL.
Figure 1: Overview of CivilBench and its capability-oriented evaluation framework for vision-language models in civil engineering.

Datasets

The benchmark consists of 972 valid samples curated from public sources (Kaggle, Elsevier, Springer) and private engineering records.

  • Category A (Sequential Reasoning): 285 samples focusing on classification, ordinal ranking, and change detection.
  • Category B (Structural Understanding): 286 samples for perception tasks like material ID and damage counting.
  • Category C (Open-world Understanding): 401 samples for complex reasoning and decision support.

The distribution of these tasks is visualized in Figure 4.

Figure 4: Sample distribution across CivilBench task categories and capability dimensions. Percentages are calculated within each category.

Key Insights and Ideas

  • The "Reasoning Bottleneck": Models are surprisingly good at "Open-world Understanding" (often >80% accuracy), likely because these tasks align well with the general knowledge and language priors they gained during pre-training. However, they hit a wall with Sequential Reasoning, particularly Ordinal Reasoning (ranking items by severity).
  • Perception $\neq$ Intelligence: A model might correctly identify a crack (perception) but fail to infer the underlying cause or the risk level (reasoning). This suggests that current VLMs are more like "advanced sensors" than "engineering assistants."
  • Numerical Fragility: Models struggle to infer rules from sparse numerical data. For example, if given four examples of earthquake data and asked to predict the risk level of a fifth, models often fail to identify the underlying mathematical relationship.
  • The Power of Priors: In supplementary experiments, the authors found that providing models with explicit "numerical priors" (formulas) and web retrieval (RAG) significantly boosted performance, but models still struggled to consistently align this external evidence with the visual task.

Key Results

The evaluation revealed a clear "difficulty hierarchy" (see Figure 7).

  • Top Performer: Gemini 2.5 Pro achieved the highest overall score (71.09% Total Accuracy), followed closely by GPT-5 (69.65%).
  • The Performance Gap: While "Open-world Understanding" saw a mean accuracy of 80.67%, "Ordinal Reasoning" plummeted to a mean of just 23.79%. Even the best model only reached 37.78% on ordinal tasks.
  • Open vs. Closed Source: Closed-source models generally lead, but open-source models like Qwen2.5-VL-7B showed impressive performance in specific categories, even tying for the lead in Open-world Understanding.
Figure 3: CivilBench task taxonomy and representative examples across all capability dimensions.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Format Constraints: The current multiple-choice format simplifies the nuance of real engineering practice.
  • Temporal Limits: The "pseudo-temporal" sequences are a start, but they don't yet capture the complexity of true time-series forecasting in structural health monitoring.
  • Future Directions: The team plans to expand the benchmark to include more diverse damage modes, regional hazards, and open-ended tasks that require models to generate detailed causal explanations rather than just selecting an answer.

CivilBench serves as a wake-up call: before we trust AI to inspect our bridges and buildings, we need to move beyond surface-level visual recognition and toward deep, physically-grounded reasoning.

Summary updated

Abstract

Artificial intelligence (AI), particularly large language models (LLMs) and vision-language models (VLMs), is increasingly applied in civil and structural engineering for disaster assessment, structural analysis, and engineering decision support. However, the extent to which current VLMs possess the reasoning capabilities required for reliable engineering judgments remains unclear. In this work, we introduce CivilBench, a comprehensive benchmark for evaluating VLMs in civil engineering and disaster-related scenarios. CivilBench covers three major task families: Sequential Reasoning, Image-based Structural Understanding, and Open-world Understanding, encompassing numerical reasoning, structural damage assessment, risk inference, causal analysis, and repair planning. The benchmark is constructed through a systematic pipeline of domain-specific data collection, engineering-oriented captioning, expert annotation, question design, and quality verification. Evaluation of representative state-of-the-art VLMs reveals substantial limitations in domain-specific reasoning, with Sequential Reasoning emerging as a major bottleneck, particularly when models must infer numerical relationships from sparse structured observations. Further experiments show that numerical priors and external knowledge can alleviate these limitations, although their effectiveness depends on the model's ability to reliably interpret and use the available evidence. CivilBench provides a capability-oriented testbed for identifying reasoning limitations and evaluating progress toward reliable AI-assisted engineering decision-making. The benchmark will be publicly available at https://huggingface.co/datasets/traceuryuan/CivilBench.
BibTeX
@misc{yuan2026civilbench,
  author = {Dan Yuan and Yue Wen and Zhili He and Zhenkui Zhou and Shicen Liu and Ziqiang Zheng},
  title = {{CivilBench: Benchmarking Vision-Language Models for Multimodal Information Integration in Civil Engineering}},
  year = {2026},
  eprint = {2611.21105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21105},
}
Figure