Diagnosing faults on an autonomous underwater vehicle is hard in a specific way. The vehicle cannot phone home while submerged, faults are rare and varied, and the hardest cases are ambiguous. A slightly nicked propeller and a drifting depth sensor can both make the telemetry look "a bit off." Today's best results on the public AUV fault benchmark come from convolutional networks that map a sensor window straight to one of five labels. Such a network cannot say why it chose a label, cannot report two faults at once, and cannot flag a fault it was never trained on. Language-model "agents", meaning several copies of a model each given a narrow job and made to check each other's work, promise that kind of explanation and flexibility. So far they have been shown only on text, code, and math, and usually without anyone measuring what they cost.
We propose a design in which each agent owns an executable physical consistency test rather than a conversational persona. One agent checks whether the depth reading agrees with the pressure reading and with integrated vertical acceleration. Another checks whether each thruster's command produces the angular response a healthy vehicle would. An adjudicating agent then combines the test outcomes into a fault hypothesis, possibly a compound one or "unknown." Everything runs on one 16 GB GPU with 4-bit open-weight models. The work is framed as a measurement study. We ask whether this decomposition beats three things: a plain classifier fed the same physics residuals, a single model given the same evidence and the same token budget, and a multi-agent system without tools. We score all of them on accuracy, calibration, unknown-fault detection, latency, tokens, and memory.
Core Claim
For small on-premise models diagnosing physical systems, any benefit of multi-agent decomposition should come from binding each role to a verifiable numeric test, not from the dialogue between roles. That benefit should appear where closed-set classifiers are weak by construction:
- held-out fault types,
- concurrent faults,
- the incipient-versus-sensor confound.
It should not appear on in-distribution accuracy, where we expect convolutional baselines to match or win. The claim can be falsified in two ways. If a hand-written rule table over the same residuals performs as well, the language models add nothing. If role-play agents without tools perform as well, the tests add nothing.
Grounding in Prior Work
The application regime is the Haizhe quadrotor AUV dataset from Ji et al.. It contains 1,225 windows over 17 channels (thruster PWM commands, 3-axis acceleration and angular rate, attitude, depth, pressure, voltage). There are five conditions: normal, slight propeller damage, severe propeller damage, depth sensor failure, and load increase. The authors themselves call its scale insufficient and publish no comparative baselines.
Wu et al. supply the discriminative reference point. They pair random-forest multiple imputation with a multi-channel fully convolutional network and report 91.4% accuracy, against 87.6% for a plain fully convolutional network. Their model is robust to up to 60% missing data. They also report that slight propeller damage is the class most often confused, and that the model cannot recognise unmodelled faults. That paper motivates our target cases and constrains our claim: we do not expect to beat it on closed-set accuracy.
Three lines of work already put language models on sensor data, and each fixes a design choice we depart from:
- Single fine-tuned model. FD-LLM LoRA-tunes one open model on encoded bearing vibration. It is strong across loads but falls to 39–58% across components. It never measures cost or calibration.
- Feedforward pipeline. BatteryAgent runs physics features, then a boosted tree with SHAP attributions, then a reasoning model. It is strictly feedforward and not profiled for constrained hardware. Its main lesson, which we reuse, is that models need a bridge from raw numbers to meaning rather than raw floats.
- Cloud reasoning model. FaultExplainer feeds PCA contribution variables to o1-preview. It reaches 8 of 11 correct or related causes on unseen Tennessee Eastman faults, but depends on a cloud model and uses only 11 test cases.
Our proposal differs from all three in the same way. The evidence each agent passes on is the pass/fail outcome of a physical hypothesis test it ran itself, on vehicle dynamics, entirely offline.
The decomposition pattern borrows method from two sources:
- LEGO splits reasoning into forward cause and backward effect analysts. In our system, "does the command explain the motion?" and "does the sensor explain the reading?" play these two roles.
- ReChisel grounds a reviewer in external tool feedback and adds an escape mechanism against endless revision loops. We copy that mechanism, and we report how often it fires.
The deployment recipe comes from two papers:
- Wolfe shows 4-bit QLoRA 7B models on a T4 matching GPT-4-Turbo on domain tasks, while noting slower inference.
- Strong et al. supply the imbalance-aware calibration metric $ECE_{Imb}$ and a hybrid confidence method that combines the model's stated confidence with a probe on its hidden states.
A known risk constrains the design. the VPP failure-query study found that stuffing retrieved context into a 1.5B model caused "cognitive confusion" and lower faithfulness. We therefore give each agent only its own test outputs, not a shared context dump.
Finally, MACM and LEGO report accuracy gains without runtime or token accounting. This is the validation gap our evaluation is built to close.
Research Program
Physics tests. First we build the consistency tests, each fitted only on normal-condition training windows.
- Depth-channel test. Because the dataset has both depth and pressure, the depth-channel agent computes the residual between them. It also computes the residual between the depth rate and the integrated vertical acceleration.
- Actuator test. The actuator agent fits a small linear input–output model from the four PWM commands to angular rates and vertical acceleration. It reports per-thruster residuals. Asymmetry across the four thrusters points to propeller damage, while a uniform shift in required command points to load increase.
- Data-quality test. A third agent reports data quality, in the form of missing-channel fraction and imputation reliance. This follows Wu et al.'s missing-data setting.
Each test returns a number, a threshold-based verdict, and a short templated sentence. This is the numeric-to-semantic bridge.
Adjudicator and loop. Second, an adjudicator agent (a 4-bit 7–8B open model, QLoRA-tuned on training windows with their test outputs) proposes a set of faults or "unknown." A reviewer agent may ask one specific test to re-run on a sub-window. Revision is capped at a fixed number of rounds, with an escape to "unknown, defer" in the style of ReChisel.
Confidence. Confidence comes from Strong et al.'s hybrid method, which combines the adjudicator's stated confidence with a probe on its hidden states.
Instrumentation. Third, we instrument everything on one 16 GB GPU. We log wall-clock latency per diagnosis, generated and prompt tokens, peak VRAM, and the number of review rounds.
New evaluation splits. We add two evaluation splits that the dataset lacks.
- Leave-one-fault-out. We train without one fault class and test whether it is flagged as unknown, and whether the correct subsystem is named.
- Composite faults. We add the depth-offset perturbation to propeller-damage windows. Ji et al. created sensor faults by additive numerical bias, so these composites follow the dataset's own sensor-fault protocol rather than an invented one. We state openly that this inherits the benchmark's limitation: bias is not a real electrical failure.
Split hygiene. Where run identifiers allow, all splits are made by run, so that windows from one run never straddle train and test. We report whether the published 80/20 split leaks across runs.
Baselines
Naive references. The naive reference is the majority-class predictor under the same splits. Its macro-F1 and $ECE_{Imb}$ show how flattering unweighted accuracy is on imbalanced data.
Strongest discriminative baseline. This is RF-MICE plus MC-FCNN from Wu et al., reported at 91.4% accuracy, with their fully convolutional network at 87.6%. We re-implement it on our run-level splits with the same hyperparameter search budget, measured in GPU-hours, as the QLoRA tuning.
For open-set detection we give it maximum-softmax and feature-distance scores. For composite faults we give it a multi-label sigmoid head. Beating it means one of two things:
- higher unknown-fault detection AUROC (area under the curve trading off correct against false unknown-flags) in leave-one-fault-out, or
- higher exact-set match on composites,
in either case without falling more than a few points below it in closed-set macro-F1. We expect a closed-set loss, and will report it.
Residual classifier (most dangerous competitor). This baseline feeds the identical physics-test residuals to a gradient-boosted tree with SHAP attributions and a distance-based unknown threshold. It is essentially BatteryAgent without the language model. We run it alongside a hand-written rule table that maps residual patterns to fault hypotheses. If either matches the agent system on the open-set and composite splits, the language-model layer is not justified.
Single-model baselines. These isolate the decomposition. One is a single QLoRA-tuned model, following the FD-LLM recipe, given all test outputs in one prompt, with chain-of-thought and self-consistency sampling. Its token budget is matched to the median the multi-agent system spends. The second is an FD-LLM-style model fed statistical features without physics tests.
Ablation of the key ingredient. The key ablation keeps the same roles and dialogue but removes tool execution, so agents see only feature summaries. It separates "tests" from "roles."
Why not just…?
Why not just use the convolutional network, which already reaches 91.4%? On in-distribution data this is probably the right answer, and we say so. The case for the agents rests on failure modes that a closed-set softmax classifier has by construction. Wu et al. note it cannot recognise unmodelled faults, and a five-way output cannot express "propeller damage plus depth bias." The leave-one-fault-out and composite splits test exactly these cases. The open-set scores we attach to the network give it a fair chance to handle them.
Why not just feed the physics residuals to a boosted tree or a rule table? This is the real threat. Most of the diagnostic signal may live in the residuals themselves. The rule table can also name hypotheses it was never trained on, if a human writes the rule. The experiment settles it directly: the same residuals go to all three systems. If the tree or the rules match the agents, the finding is that on-premise language models are unnecessary for this vehicle. That is a useful negative result, given how often such systems are proposed without measurement. The agents could win in two places. One is composite and borderline cases, where a reviewer requesting a targeted re-test on a sub-window changes the verdict. The other is calibrated deferral.
Why not just one fine-tuned model with chain-of-thought? MACM and ReChisel report large gains from role separation, but on math and code with frontier models. MACM reports only a 3–5% gain for open-source LLaMA-3, which suggests that the gain may shrink for small local models. Our token-matched single-model baseline tests whether decomposition helps at 7B scale, once the extra compute is paid for fairly.
Proposed Experiments
Splits. All systems are evaluated on the Ji et al. dataset under three splits: run-level closed-set, leave-one-fault-out (four folds), and synthetic composites. Each is repeated over five seeds.
Metrics. We report:
- macro-F1 and per-class recall, with slight propeller damage singled out;
- unknown-fault AUROC;
- composite exact-set match;
- $ECE_{Imb}$ and selective accuracy as a function of deferral rate;
- latency per diagnosis, tokens, peak VRAM, and loop-escape rate.
All are reported on the same single-GPU hardware. We also stress-test missing data at 20/40/60% to compare against Wu et al.'s robustness regime.
Decisive checks. Three comparisons decide the outcome:
- Agents versus residual tree and rule table on the open-set and composite splits. This decides whether language models earn their cost.
- Tool-free agents versus tool-bound agents. This decides whether tests, rather than roles, carry the gain.
- Latency and token cost against the vehicle's surfacing interval. A diagnosis slower than the time the vehicle can afford to wait is not deployable, however accurate.
There are no reported numbers on Lacuna for any language-model or open-set method on this dataset, so every result beyond Wu et al.'s closed-set figures will be new. We will release them as the first multidimensional baseline suite for the benchmark.
References
- Autonomous underwater vehicle fault diagnosis dataset
- Fault Diagnosis of Autonomous Underwater Vehicle with Missing Data Based on Multi-Channel Full Convolutional Neural Network
- Large language models for explainable fault diagnosis of machines
- BatteryAgent: Synergizing Physics-Informed Interpretation with LLM Reasoning for Intelligent Battery Fault Diagnosis
- FaultExplainer: Leveraging large language models for interpretable fault detection and diagnosis
- LEGO: A Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for Causality Explanation Generation
- ReChisel: Effective Automatic Chisel Code Generation by LLM with Reflection
- Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
- Trustworthy and Practical AI for Healthcare: A Guided Deferral System with Large Language Models
- Implementation of a Device Failure Query Model for the Virtual Power Plant Smart Operation and Maintenance Platform Based on Retrieval-Augmented Generation Technology
- MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems