Moving Generality from Model to Modeler: LLM Agents for PHM on a Deployment-Style Benchmark

Zhenling Chen, Zhiguo Zeng, Anne BarrosarXiv 2026 · arXiv

TL;DR

Generality in AI is shifting from the model to the modeler. This paper demonstrates that LLM-based coding agents can outperform traditional AutoML and even state-of-the-art human-designed models in Prognostics and Health Management (PHM) by building custom, machine-specific predictors on the fly.

What Problem Does This Paper Address?

In engineering fields like PHM, the goal is to predict when a machine—like a jet engine or a wind turbine—will fail. The "holy grail" has long been a general model that works across different types of equipment. However, every machine is slightly different, and a model that works perfectly for one bearing might fail on the next one due to different operating conditions or sensor layouts.

The authors ask a provocative question: Should we stop trying to build general models and instead build general "modelers"? If an AI agent can look at a specific machine's data and write a custom algorithm for it in two hours, does the model itself still need to be "general"?

Methods

The researchers introduced a new evaluation framework called NextUnit-Bench. Unlike standard benchmarks that test a static model on a fixed test set, this is a "deployment-style" benchmark:

  1. The Sandbox: An LLM agent is given a task description and raw data from "development" machines. It has two hours to write, test, and debug a custom predictor.
  2. The Seal: The agent delivers a "sealed" executable program.
  3. The Test: This program is run on "hidden units"—machines the agent has never seen.
Fig. 1. Deployment-style evaluation. The agent sees the task description and the development units, works for two hours in a sandbox, and delivers a sealed, executable predictor. The evaluator runs it offline on each hidden unit; the hidden units, their labels, the scoring code, and the score itself remain invisible to the agent.

To understand why agents succeed, they performed "plan transplants." They had a strong agent (Astra) write a plain-text design plan (no code) and handed it to a weaker agent (Luna) to implement. This isolated whether the advantage came from better coding skills or better design decisions.

Datasets

The study used NextUnit-Bench, which aggregates 24 public PHM datasets (split into 58 folds). These cover:

  • Assets: Bearings, gearboxes, milling cutters, battery cells, turbofan engines, oil wells, and more.
  • Tasks: Remaining Useful Life (RUL) prediction, fault classification, and anomaly detection.
  • The Split: Crucially, data is split by physical unit or operating condition, ensuring the test machines are truly "unseen."

Key Insights and Ideas

  • Generality is Moving: The "intelligence" isn't in a one-size-fits-all neural network; it's in the agent's ability to adapt its modeling strategy to the specific data in front of it.
  • The Plan is the Secret Sauce: Most of the performance gap between a "frontier" agent (like GPT-4 class models) and a weaker one lies in the design plan. When a strong agent followed a weak agent's plan, its performance dropped to the weak agent's level.
  • Data Trumps Domain Toolkits: Surprisingly, giving agents specialized PHM "toolkits" or expert-written libraries didn't help. The strongest agents gained their edge by learning directly from the raw data rather than relying on pre-packaged domain knowledge.
  • The Reproducibility Gap: The authors found that many published PHM accuracies rely on specific "conventions" (like capping predicted life at 125 cycles). When these methods are implemented for real-world deployment without those shortcuts, their error rates can double.
  • Capability as "Strength": Local, open-weight models (like Qwen or Gemma) aren't necessarily bad at designing models, but they struggle to execute them. They often get stuck in coding loops or run out of time, a phenomenon the authors compare to "stress-strength" interference in reliability engineering.

Key Results

  • Agents vs. The World: Frontier agents removed nearly half of the baseline error ($S \approx 43\%$), while AutoML systems reached less than half of that score ($S < 20\%$).
  • Beating the Literature: In a head-to-head comparison on four major datasets, the agent's custom-built designs beat 24 out of 30 faithful implementations of published research papers.
  • The Power of Customization: On an air-handling unit (AHU) dataset, agents discovered a "shortcut" in how the data was collected (a shared schedule). While a "general" model couldn't know this, the agent used it to achieve a high score—exactly what a human engineer would do on-site.
Fig. 2. Score 𝑆over the 24 datasets, with 95% intervals from a bootstrap over datasets. Frontier agents remove a third to nearly half of the baseline’s error; local open-weight agents, AutoML, even with generic features, and time-series foundation models (TSFM) fine-tuned on each fold remain below 20. AutoML was not run on MIMII DUE and scores zero there; the fixed random forest is one of the two models that make up the baseline.

Limitations and Future Work

  • Data Leakage: Since the datasets are public, the agents might have "seen" them during their pre-training, though they weren't told the names of the datasets during the tests.
  • The "Cold Start" Problem: The benchmark assumes you have some labeled data from similar machines. If a brand-new type of machine arrives with zero labels, the agent has nothing to "learn" from.
  • Local Deployment: Currently, models small enough to run on a factory's local hardware (local LLMs) are much less reliable at building these custom predictors than giant "frontier" models accessed via API. Closing this gap is a key next step for industrial privacy.

Summary updated

Abstract

Coding agents built on large language models (LLMs) now help develop many models published as general solutions. Should generality instead reside in the agent that builds a model for each new task? We examine this question in prognostics and health management (PHM), where every model must work on unseen equipment. Our deployment-style benchmark, NextUnit-Bench, covers 24 public datasets split into 58 folds that hold out units or operating conditions; each agent has two hours to deliver an executable predictor, then run on the hidden units. First, frontier agents outperform AutoML, which reaches less than half the best agent's score, and faithful implementations of published methods on four datasets, of which only 6 of 30 beat the strongest agent's own design; published accuracies, where reproducible, rest on the papers' own conventions, and a single convention, capping the remaining life of test engines, can halve the reported error. Second, the agents' advantage is a design made on each machine's data; once made, a plain-text plan of a few hundred words carries most of the gap between two frontier agents, and given a weaker agent's plan, the strongest agent falls to that agent's level, often to identical predictions. Before touching the data, however, it matters little whose plan an agent carries out, and domain toolkits appear to help neither: what sets the strongest apart is how much it learns from the data. Third, such designs reach only implementers that can execute them: given the strongest agent's designs, a locally deployable open-weight model delivers half as often, even when they are lightened, and scores no higher than without them. Generality, we argue, is not disappearing but moving, from the model to the modeler.
BibTeX
@misc{chen2026moving,
  author = {Zhenling Chen and Zhiguo Zeng and Anne Barros},
  title = {{Moving Generality from Model to Modeler: LLM Agents for PHM on a Deployment-Style Benchmark}},
  year = {2026},
  eprint = {2611.25252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25252},
}
Figure