IndustryLLM: Failure-Driven LLM Training for Industrial Procurement

Liang Ding, Zhiang Xu, Yuyang Sheng, Bin Chen, Songlin Bai, Run Zhu, Dingjun Wu, Hui Xu, Yandi Wang, Fulin Shi, Leilei Gan, Linlin Yu, Qihuang Zhong, Keqin Peng, Yalong Li, Chengfu Huo

TL;DR

IndustryLLM is an open-weight 35B Mixture-of-Experts (MoE) model specialized for the high-stakes world of industrial procurement. By employing a "failure-driven" training recipe that targets specific domain gaps and enforcing a strict "three-valued logic" for product eligibility, the model delivers substantial gains in transaction volume (GMV) and a 4x reduction in latency compared to general-purpose baselines.

What Problem Does This Paper Address?

Industrial procurement is a minefield of technical jargon, ambiguous standards, and zero-tolerance safety requirements. When a buyer asks for a "42-luo-mu bolt" (a colloquialism for 42CrMo steel) or cites a truncated standard like "16674," general-purpose LLMs often stumble.

The authors identify three core challenges:

  1. Register Mismatch: The gap between formal engineering standards (dry, technical) and buyer inquiries (informal, messy).
  2. Factual Brittleness: The tendency of models to hallucinate technical specs or rely on obsolete standards.
  3. The "Satisfied by Default" Fallacy: In safety-critical engineering, if a product record doesn't mention a required temperature rating, a model shouldn't assume it's fine. It must treat missing information as a hard "Unknown."

Methods

The researchers started with Qwen3.5-35B-A3B-Base and applied a two-stage adaptation process:

1. Failure-Driven Continued Pre-training (CPT)

Instead of just feeding the model more data, they reconstructed a 20B-token subset of their corpus to fix specific failure modes:

  • Multi-register Rewriting: They used a teacher model to recast formal standards into 10 different genres (e.g., troubleshooting notes, buyer-seller dialogues) and 8 writing styles to bridge the jargon gap.
  • Confidence-Routed Minimal Editing: An automated pipeline identified factual errors in industrial web text (like expired standards) and corrected them via localized "string diffs" only when high-confidence evidence was available.
  • Weakness-Targeted QA Synthesis: They generated synthetic Q&A pairs specifically for topics where the model showed high failure rates during intermediate testing.

2. Supervised Fine-Tuning (SFT)

The model was trained on teacher-generated targets (from Qwen3.5-Max) that were strictly filtered for factual correctness and reasoning density. They released two variants: Think (retains the chain-of-thought for complex reasoning) and No-Think (optimized for sub-second latency).

3. The Evidence-Gated Interface

To prevent dangerous hallucinations, they formalized a deployment interface using three-valued logic: Satisfied, Violated, or Unknown. Crucially, "Unknown" is never treated as "Satisfied," triggering a request for clarification rather than a potentially catastrophic product recommendation.

Datasets

  • Training Corpus (~100B tokens): Includes 5B tokens of national standards (GB/T) and technical archives, 10B tokens of de-identified real-world transaction records, 25B tokens of filtered industrial web text, and 60B tokens of general-domain replay to prevent catastrophic forgetting.
  • IndustryBench: A diagnostic benchmark consisting of 2,049 expert-curated questions used to measure domain mastery and safety.
  • General Benchmarks: MMLU-Pro, GSM8K, HumanEval-Plus, etc., were used to ensure the model retained general reasoning capabilities.

Key Insights and Ideas

  • Linguistic Diversity Matters: The "aha moment" in their CPT strategy was that rewriting formal knowledge into diverse registers (like blogs or chats) significantly improved the model's ability to understand real-world buyers without losing technical accuracy.
  • Sparse Efficiency: By using a Mixture-of-Experts (MoE) architecture (activating only ~3B parameters per token), they achieved domain performance that surpassed much larger dense models, making it viable for production environments.
  • Demarcation of Knowledge: The authors argue for a strict boundary: the LLM should structure the intent (what the buyer needs), but the eligibility (whether the product works) must be determined by an auditable system checking fresh, non-parametric evidence.

Key Results

  • Production Impact: In randomized online A/B trials (N ≈ 100,000 users/day), IndustryLLM drove a +4.25% increase in Gross Merchandise Value (GMV) and an +8.3% increase in satisfied inquiries.
  • Latency: End-to-end response time dropped from 6–7 seconds to 1.5 seconds, a 4x improvement essential for e-commerce.
  • Offline Accuracy: The model saw a +2.97 percentage point gain in exact match for query structuring tasks compared to the base model.
  • Safety: On the IndustryBench safety-adjusted metric, IndustryLLM (35B) outperformed significantly larger models, including the 397B parameter version of Qwen3.5.

Limitations and Future Work

  • Geographic/Language Scope: The current study is centered on the Chinese B2B market. The authors note that cross-lingual transfer and international regulatory compatibility remain unverified.
  • Multimodal Gap: While the base model has vision capabilities, the vision encoder was kept frozen for this study. Future work could involve extracting constraints directly from CAD blueprints or technical drawings.
  • Model Lineage: The reliance on the Qwen family for both generation and judging introduces potential "intra-family" biases that could be explored with more diverse model ensembles.

Availability: The model weights and inference configurations are publicly available on Hugging Face.

Summary updated

Abstract

Industrial procurement requires language models to bridge informal buyer jargon, sparse marketplace attributes, and authoritative engineering standards under strict safety tolerances. We present IndustryLLM, an open-weight industrial language model trained from Qwen3.5-35B-A3B-Base (35B total parameters with ~3B activated per token, with the vision encoder frozen). Rather than relying on generic text scaling, we introduce a failure-driven adaptation recipe spanning continued pre-training (CPT) and supervised fine-tuning (SFT). CPT leverages a curated ~100B-token corpus integrating 5B tokens of national standards (e.g., GB/T) and technical archives, 10B tokens of de-identified real-world industrial transaction and inquiry records, and 60B tokens of general replay. To overcome register mismatch and factual brittleness, we systematically reconstruct an estimated 20B-token domain subset via multi-register rewriting across 10 genres and 8 writing styles, confidence-routed minimal factual editing, and error-targeted QA synthesis (resolving colloquial typos like '42-luo-mu' -> 42CrMo, expanding ambiguous codes like '16674' -> GB/T 16674, and clarifying conflicting dimensional specs). For downstream deployment, we formalize an evidence-gated constraint-evaluation interface enforcing three-valued logic where unverified product evidence remains unknown rather than satisfied. Offline evaluations demonstrate consistent gains on procurement-query structuring (+2.97 percentage points in exact match, 95% CI [2.11, 3.86] in No-Think mode), while randomized online A/B experiments in production yield substantial improvements (+4.25% GMV, +8.3% satisfied inquiries) alongside a latency reduction from 6-7 s to 1.5 s. Model weights and configs are released at https://huggingface.co/alibaba-multimodal-industrial-ai/IndustryLLM.
BibTeX
@misc{ding2026industryllm,
  author = {Liang Ding and Zhiang Xu and Yuyang Sheng and Bin Chen and Songlin Bai and Run Zhu and Dingjun Wu and Hui Xu and Yandi Wang and Fulin Shi and Leilei Gan and Linlin Yu and Qihuang Zhong and Keqin Peng and Yalong Li and Chengfu Huo},
  title = {{IndustryLLM: Failure-Driven LLM Training for Industrial Procurement}},
  year = {2026},
  eprint = {2609.31871},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.31871},
}
Figure