Most automated fault diagnosis systems quietly assume that when something goes wrong, exactly one thing went wrong. This is the single-fault assumption. It is convenient because a system with $N$ components then has $N$ possible explanations to check instead of $2^N$. It is also what the available training data supports, since test benches record one seeded defect at a time. The danger comes when two faults occur together. A loud fault can mask a quiet one, and two vibration signatures can interfere. In those cases a single-fault classifier does not become unsure. It confidently names one fault and misses the other. Selective prediction, meaning a model that may abstain and hand the case to a human instead of answering, is the natural remedy. But existing abstention rules key on low confidence, and masking produces high confidence.
We propose to turn the single-fault assumption into a statistical test that runs on every input. The diagnoser asks whether any single-fault hypothesis, or the healthy hypothesis, explains the observation well enough. It calibrates that question with conformal p-values, which are rank-based scores computed against held-out data and carry finite-sample error guarantees. If no single explanation is adequate, the system abstains with a specific message: "more than one thing may be wrong here." Where faults can be removed from the data, as with a broken sensor whose stream can be masked, the system goes further. It commits to the faults it can isolate and repeats the test on what remains. The output is a partial diagnosis, a set of confirmed faults plus a calibrated flag for unexplained residue. The false-flag rate is guaranteed for every single-fault class, and the method needs no training data for compound faults.
Core Claim
The claim has two parts. First, a selection score built from the best single-fault explanation will detect masked compound faults much more reliably than confidence-based abstention or unstructured out-of-distribution scores, because masking hides a secondary fault from the classifier's confidence but not from the fit of the best explanation. Second, this score can be calibrated using only healthy and single-fault data, which is the data practitioners actually have, so that the false-alarm rate on genuine single-fault or healthy inputs is at most a chosen level $\alpha$ within each fault class.
Concretely, let $r_j(x)$ measure how badly hypothesis $j$ explains input $x$, for $j = 0$ (healthy) through $N$ (each single fault). An example is the reconstruction error of the remaining channels after masking or modelling component $j$ as faulty. For each class $j$ we hold out calibration windows $x^{(j)}_1,\dots,x^{(j)}_{n_j}$ drawn from that class and define the per-class p-value
$$p_j(x) = \frac{1 + \#\{i : r_j(x^{(j)}_i) \ge r_j(x)\}}{n_j + 1}.$$
The system raises a compound-fault flag when $\max_j p_j(x) \le \alpha$, that is, when every single explanation is rejected. If $x$ truly comes from class $j$ and is exchangeable with that class's calibration data, then $P(p_j(x) \le \alpha) \le \alpha$, so the flag fires on class-$j$ inputs at most an $\alpha$ fraction of the time, separately for every $j$.
This per-class guarantee matters. Marginal calibration, which only controls error averaged over all inputs, hides collapses on rare classes. Act or Defer showed this directly. Marginal conformal selection on a rare clinical outcome achieved only about 10% correct actions for the positive class at a nominal $\alpha = 0.10$, while class-stratified selection kept coverage above 99%.
Research Program
The work proceeds at two levels of ambition, matched to how faults physically behave.
The first level is detection only: is the single-fault assumption violated? This works for any system, including ones where a fault physically perturbs the whole signal and cannot be "removed." Bearings and battery cells are examples. Here the system outputs either a calibrated single-fault diagnosis or an abstention flagged as possible compound fault.
The second level is sequential isolation. It applies where a fault can be cleanly masked, as with a sensor that is stuck, drifting, or spiking. This level reuses the "iterative isolate-and-continue" loop from Tureis. Tureis is a BERT-style masked-reconstruction transformer for smart-home sensors that, once it flags a sensor as faulty, masks it out and re-examines the rest. Tureis decides each step with heuristic residual thresholds and offers no error control. We replace each step with a conformal test.
The test works as follows. After committing to a faulty set $S$, we ask whether the remaining sensors look healthy given that $S$ is masked. We calibrate that test on healthy windows with $S$ artificially masked. Masking healthy data is free, so the null distribution for every $S$ the procedure might reach is available without any compound-fault data. The loop stops at the first test that does not reject. Each committed fault was declared only after a level-$\alpha$ rejection, and we will bound the total over-counting error across steps by splitting the error budget across the step depth.
This design draws on three prior lines of work, each in a different role. The partial-abstention theory of Nguyen and Hüllermeier derives per-label reject rules for multi-label classification. It supplies the motivation that answering part of a label vector beats rejecting the whole instance. It also constrains us, because its rules assume conditionally independent labels, which is exactly what masking breaks. The label-wise clinical abstention of Vazhentsev et al. supports the same partial-answer view empirically, though on static text and without guarantees. The stratified conformal selection of Act or Defer supplies the per-class calibration technique, which we move from single-label clinical prediction to a union-of-hypotheses test over fault classes.
The physical-coupling case carries a caveat that the experiments must confront. In batteries and turbines, a fault cannot be masked away. The hybrid physics/deep model on C-MAPSS documents degradation "smearing" across correlated channels, which motivates using a physics-informed residual for $r_j$ where one exists. The guarantee also depends on exchangeability between calibration and deployment data. Abburi et al. show this is routinely violated in condition monitoring: an SVM's macro-F on XJTU-SY falls from 0.779 to 0.309 when random splits are replaced by bearing-wise splits. We therefore treat validity under machine-wise splits as an empirical question, not an assumption.
Baselines
The naive reference point is a forced-choice single-fault classifier: always answer with the arg-max class. For it we report the "confidently wrong" rate, the fraction of compound-fault windows assigned a single label with top-class probability above 0.9. This rate quantifies the masking problem the proposal targets. If it is low, confidence-based abstention would already suffice and the motivation weakens.
The confidence-abstention baseline is the softmax-response selective classifier of Geifman and El-Yaniv. It thresholds top-class confidence to meet a target risk. We also report an entropy-threshold variant. Both run on the same backbone and calibration split as our method, with thresholds set to the same per-class false-flag level $\alpha$.
The strongest existing multi-fault isolator is Tureis itself, run with its published heuristic thresholds and then with thresholds retuned on our calibration split, so that it gets the same tuning budget. Tureis reports localization-F1 gains of +7.6% to +25.0% for single failures and up to +35.4% for up to five concurrent failures over ThingsDND and DICE, in Tureis. The file gives only these relative gains, not absolute F1. Our method beats it if two things hold together. Its per-class false-flag rate must stay at or below $\alpha$, where Tureis's thresholds carry no such control. And its committed-fault isolation F1 on multi-failure windows must be at least equal at matched coverage.
The strongest generic competitor is conformal out-of-distribution detection. This uses a nearest-neighbour or Mahalanobis distance in the backbone's embedding space, calibrated per class on the same single-fault data. It is the real threat because compound faults are, by definition, out of distribution relative to single-fault training.
Two ablations isolate the key ingredient. The first replaces the min-over-hypotheses score with the unstructured full-reconstruction error. The second replaces per-class calibration with a single marginal threshold. Monte Carlo dropout, which reruns a network many times with random units switched off and uses the spread of outputs as uncertainty, is included only as a cost and quality reference. On MIMIC-IV coding it performed worse than random rejection (Vazhentsev et al.), and it multiplies the 1.5–4 ms per-window edge latency Tureis reports by the number of passes.
For error-controlled compound-fault detection on any of these benchmarks, there is no reported result found on Lacuna. The comparison must therefore be run, not quoted.
Why not just…?
Why not just threshold the classifier's confidence? Masking predicts that the dominant fault drives confidence up, not down. If that holds, softmax or entropy abstention will pass compound faults through as confident single-fault answers. The forced-choice "confidently wrong" rate on multi-failure windows tests this directly. If confidence thresholding matches our method's detection power at equal per-class $\alpha$, the structured score is unnecessary, and we will report that.
Why not just use a calibrated out-of-distribution detector? This is the serious alternative, and it may work well when two faults sum to something far from either alone. Our argument is that a masked secondary fault leaves the embedding close to the dominant fault's cluster. The residual left after the best single explanation, however, still carries the secondary fault's trace. The decisive comparison is detection power stratified by severity ratio between the two faults: an out-of-distribution score should lose most where one fault dominates. Even if detection power ties, conformal OOD outputs only "something is unusual." It cannot commit to the isolated faults and flag the rest, which is the partial diagnosis our second level provides. If it ties on both detection and downstream utility, the contribution shrinks to the evaluation protocol, and we would say so.
Why not just synthesize all fault combinations and train a multi-label model? Synthesis is genuinely useful, and we include a multi-label model trained on synthetic pairs with per-label conformal thresholds as a further baseline. It cannot cover $2^N$ combinations, though. Generators also miss regime structure: Battery Fault reports a 20% performance drop for traditional models moving from charging to discharging data. We test the synthetic model on combinations held out of its synthesis set and under regime shift. Our method needs no combination data, so it should degrade less on held-out combinations.
Proposed Experiments
The primary testbed is the smart-home suite from Tureis: HouseA, HH102, Tulum, Atmo1, and Tokyo, with injected stuck-at, drift, noise, and spike faults. We calibrate on healthy and single-sensor-fault windows and test on one to five concurrent faults. Masking is exact here, so both levels of the method apply.
The secondary testbed is CH-BatteryGen from Battery Fault, with 1,000 vehicles and a reported CNN fault-classification F1 of 0.9280 in LFP discharging. We use its equivalent-circuit injection to create held-out dual-fault segments, such as self-discharge combined with high resistance, which exercises the detection-only level under physical coupling.
The third testbed checks validity under machine shift. CWRU and XJTU-SY are evaluated with bearing-wise splits as Abburi et al. recommend, and calibration data comes from bearings disjoint from the test bearings. We measure whether the per-class false-flag rate stays at $\alpha$ when exchangeability is strained. Artificially superposed CWRU signals serve only as a clearly labeled stress test, since real interference is nonlinear.
The metrics are as follows. We report per-class empirical false-flag rate against nominal $\alpha$; detection power on compound faults, broken down by number of faults and by severity ratio; and isolation F1 of committed faults with coverage. We also report risk-coverage curves and edge latency on a Raspberry Pi-class device.
Three checks decide the outcome. Per-class false-flag rates must sit at or below $\alpha$ on in-distribution splits, with any violation under bearing-wise splits quantified. The structured score must beat both confidence thresholding and conformal OOD on dominant-fault compound cases. The sequential level must recover at least Tureis's multi-failure isolation F1 while keeping its error control.
References
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- Tureis: Transformer-based Unified Resilience for IoT Devices in Smart Homes
- Reliable Multi-label Classification: Prediction with Partial Abstention
- Uncertainty-aware abstention in medical diagnosis based on medical texts
- Hybrid deep fault detection and isolation: Combining deep neural networks and system performance models
- A Closer Look at Bearing Fault Classification Approaches
- Selective Classification for Deep Neural Networks
- Battery Fault: A Comprehensive Dataset and Benchmark for Battery Fault Diagnosis