An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios

Zongjie Li, Wenying Qiu, Pingchuan Ma, Yichen Li, You Li, Sijia He, Baozheng Jiang, Shuai Wang, Weixi GuarXiv 2024 · DOI · arXiv

Introduction

While Large Language Models (LLMs) have demonstrated remarkable capabilities in general benchmarks, their application in heavy industry—such as mining, power grids, and petrochemicals—remains largely unexplored. In these sectors, precision and reliability are paramount; a "hallucination" can lead to significant safety hazards or multi-million dollar operational errors. This paper presents the first comprehensive evaluation of LLMs specifically within Chinese industrial contexts, testing 10 prominent models across eight specialized sectors.

Methodology

The researchers developed a two-pronged evaluation framework to assess both the knowledge and the reliability of models like GPT-4, Ernie, and Spark.

1. Accuracy Testing

A manually curated benchmark of 1,200 domain-specific problems was derived from national authoritative question sets, technical training guidelines, and production standardization conditions. These cover eight key sectors: Electronic Equipment, Equipment Manufacturing, Iron & Steel, Mining, Power, Petrochemical, Building Materials, and Textiles.

2. Metamorphic Testing (MT)

To measure robustness, the researchers created 13,631 variants of the original "seed" questions. By perturbing inputs—such as changing units (100cm to 1m), swapping multiple-choice option orders, or adding irrelevant distractors—the study quantified how consistently a model maintains its correct answer despite minor, non-semantic changes.

Key Results and Performance Gaps

The study reveals that while LLMs show promise, they are not yet ready for autonomous industrial deployment.

Low Overall Accuracy

No model achieved an accuracy score higher than 60% across the industrial sectors. GPT-4 led the evaluation with a score of 0.59, followed by GPT-3.5 and local Chinese models. The performance floor was significantly lower, with some anonymized local models scoring as low as 0.33.

Figure 2: Accuracy comparison across 10 evaluated LLMs showing GPT-4 as the top performer.

Robustness Disparity

Robustness varied significantly by model and sector. Global models (GPT series) demonstrated higher robustness in logic-related perturbations, while local models often struggled with consistency when question formats were altered. The "Equipment" sector showed the highest average robustness (0.72), whereas the "Power" sector proved the most challenging (0.66) due to its complex technical standards.

Figure 3: Robustness scores across eight industrial sectors, highlighting the performance gap between models.

Key Insights

  • Reasoning vs. Terminology: A distinct trade-off exists between global and local models. Global models excel at mathematical reasoning and logical consistency. Conversely, local Chinese models often perform better with hyper-specific industrial jargon and localized regulatory nuances.
  • The "Order" Fragility: A surprising finding was the sensitivity of models to option ordering. Simply moving a correct answer from 'A' to 'B' frequently caused models—especially local ones—to fail, suggesting a reliance on pattern matching over deep comprehension.
  • Self-Knowledge Blind Spots: GPT-4 exhibited a unique form of "self-awareness," occasionally refusing to answer when it lacked specific data rather than hallucinating a response. This trait is critical for responsible industrial AI where "I don't know" is safer than a wrong guess.
  • Robustness-Accuracy Correlation: The data suggests that models with higher accuracy tend to be more robust. True understanding of the underlying physics or logic makes a model less susceptible to unit changes or distractors.

Limitations and Future Work

The study notes that LLMs are evolving rapidly, and the versions tested (e.g., GPT-4-0613) may be superseded. Furthermore, the evaluation used a deterministic setting (temperature = 0), which may not reflect conversational or creative industrial applications. Future research will expand into testing efficiency (latency/cost), data privacy, and security against adversarial attacks in industrial code generation.

Summary updated

Abstract

Recent years have witnessed the rapid development of large language models (LLMs) in various domains. To better serve the large number of Chinese users, many commercial vendors in China have adopted localization strategies, training and providing local LLMs specifically customized for Chinese users. Furthermore, looking ahead, one of the key future applications of LLMs will be practical deployment in industrial production by enterprises and users in those sectors. However, the accuracy and robustness of LLMs in industrial scenarios have not been well studied. In this paper, we present a comprehensive empirical study on the accuracy and robustness of LLMs in the context of the Chinese industrial production area. We manually collected 1,200 domain-specific problems from 8 different industrial sectors to evaluate LLM accuracy. Furthermore, we designed a metamorphic testing framework containing four industrial-specific stability categories with eight abilities, totaling 13,631 questions with variants to evaluate LLM robustness. In total, we evaluated 9 different LLMs developed by Chinese vendors, as well as four different LLMs developed by global vendors. Our major findings include: (1) Current LLMs exhibit low accuracy in Chinese industrial contexts, with all LLMs scoring less than 0.6. (2) The robustness scores vary across industrial sectors, and local LLMs overall perform worse than global ones. (3) LLM robustness differs significantly across abilities. Global LLMs are more robust under logical-related variants, while advanced local LLMs perform better on problems related to understanding Chinese industrial terminology. Our study results provide valuable guidance for understanding and promoting the industrial domain capabilities of LLMs from both development and industrial enterprise perspectives. The results further motivate possible research directions and tooling support.
BibTeX
@misc{li2024empirical,
  author = {Zongjie Li and Wenying Qiu and Pingchuan Ma and Yichen Li and You Li and Sijia He and Baozheng Jiang and Shuai Wang and Weixi Gu},
  title = {{An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios}},
  year = {2024},
  eprint = {2402.01723},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.01723},
}
Figure