Solver-in-the-Loop Delphi for Eliciting Repair Rules

When an autonomous system breaks a rule, something else has to govern what it does next. A drone blown into restricted airspace should broadcast its position, leave by the shortest safe route, and still be recorded as having violated airspace. Logicians call these follow-up duties contrary-to-duty obligations: obligations that only switch on once a primary obligation has been breached. Run-time "normative supervisors" can now enforce such rules over a learned policy, picking the least-bad action when every option breaks something. But the penalty weights and priority orderings that decide what counts as least bad are written by the system's programmers. The established way to get defensible rules from domain experts is the Delphi method: several anonymous rounds of questionnaires, where each round shows panelists the group's statistics and invites them to revise. Delphi, however, produces natural-language statements and rating distributions, not executable rules. It also never checks whether the agreed statements, once combined, produce sensible behavior in the awkward situations where they collide.

This note proposes a Delphi process with the logic solver inside the loop. Between rounds, the panel's current consensus is compiled into a weighted rule program. The solver then finds the concrete breach scenarios where that program's chosen repair action is most fragile, meaning a small change in the weights would flip the decision. Those scenarios, together with the action the current consensus would actually take, become the next round's questionnaire. Panelists stop rating abstract severities and start judging what the system would do. They revise, and the weights are refit from their judgments. The bet is that experts' abstract severity ratings do not reliably encode their own judgments about concrete repairs. If that is right, showing the solver's consequences closes the gap faster and with fewer judgments than classical Delphi or a one-shot scenario survey.

Core Claim

The claim is narrow and testable. For a fixed number of expert judgments, a Delphi process whose items are chosen by the solver and whose feedback shows each panelist the solver's implied repair action will yield weights and priority levels that better predict an independent expert panel's choices on held-out breach scenarios. Two comparisons define "better": abstract-rating Delphi, and fitting weights to a one-shot batch of randomly sampled scenario choices.

A second, contrary-to-duty-specific claim concerns structure rather than numbers. For each primary/repair pair, the process will also elicit whether the breach cancels the primary duty or merely overshadows it, so that the violation stays on record. Getting this right measurably reduces norm erasure: cases where the encoded system silently forgets that a violation occurred.

Where This Sits

The enforcement machinery is reused, not reinvented. Deontic Paradoxes in ASP with Weak Constraints encodes obligations as weak constraints, rules that can be broken at a cost. Each carries a weight $w$ and a priority level $p$, and the solver searches for the cheapest consistent outcome. This resolves Ross's paradox, Plato's dilemma and the fence scenario without the logical explosion that classical deontic logic suffers. A Normative Supervisor for Reinforcement Learning Agents and Enforcing ethical goals over reinforcement-learning policies wrap such reasoning around a black-box reinforcement-learning policy, including a contrary-to-duty "Passive Vegan" Pac-Man rule.

Both lines supply the method this proposal reuses, and both leave the same input unexamined: the weights and superiority orderings are hand-set. The distinction between cancelling and overshadowing comes from Cancelling and Overshadowing. Here it becomes something the panel decides per rule pair rather than a fixed design choice.

On the consensus side, Delphi studies such as Dombrowski et al. and Byrne et al. define agreement through rating stability and interquartile-range shrinkage across rounds. Studies such as Vu et al. use content-validity ratios. None of these checks whether agreed items jointly entail acceptable behavior, which is precisely the gap a solver can fill. CALMA motivates the participatory angle: its cohorts derived alignment axes that generic categories missed. It stops at qualitative labels, with no executable rules.

The closest-named prior work is the Delphi moral model of Delphi: Towards Machine Ethics and Norms and Investigating machine moral judgement through the Delphi experiment. Despite the name, it is a T5-11B model fine-tuned on 1.7 million static crowd judgments, with a MAX-SAT layer for consistency. It judges single actions, runs no multi-round expert process, and has no notion of an obligation triggered by a prior breach.

One result constrains the design. Mitigating Adversarial Norm Training with Moral Axioms shows that purely bottom-up norm learners adopt inverted moral labels 100% of the time. For that reason the proposal keeps a small set of hard constraints, the perfect duties such as "do not cause injury", outside the panel's reach. Only the weights among the soft, tradeable duties are elicited.

Research Program

The domain is the drone archetype, since aviation has a real expert population: operators, regulators and safety engineers. It is rendered as a gridworld simulator built on the same supervisor architecture as the Pac-Man testbeds. The rule base has roughly fifteen to twenty-five soft rules. Primary duties include airspace limits, privacy buffers and battery margins. Repair duties include broadcast, exit route, landing and operator notification. A handful of hard constraints sit alongside them. A rule base this small also makes the exact conflict-contribution measure $I_\wp$ of Raddaoui et al. affordable: its $2^n$ cost is a real barrier at scale but not at $n \approx 25$. It is used to tell panelists which agreed rules contribute most to any outright inconsistency.

The panel's judgments are linked to the weights through a standard choice model. Write $s$ for a breach scenario, $a$ for a candidate action, $v_i(s,a) \in \{0,1\}$ for whether action $a$ violates soft rule $i$ in $s$, and $w_i$ for its weight within a priority level. The probability that a panelist endorses $a$ is then modeled as

$$P(a \mid s, w) \propto \exp\Big(-\beta \sum_i w_i\, v_i(s,a)\Big),$$

where $\beta$ is a fitted rationality constant. Weights are fit by maximum likelihood over all scenario judgments to date. Priority levels are assigned where the fit says one rule's weight dominates every feasible sum of lower rules. In those cases the panel is asked explicitly whether the trade-off is truly forbidden, which is a lexicographic level, or merely steep.

Each round proceeds as follows. The solver samples candidate breach scenarios from the simulator. It scores each by decision fragility, defined as the probability under the current weight posterior that the cost-minimizing action changes. It presents the ten to fifteen most fragile scenarios. Each scenario is shown with the action the current consensus would take, the rules that action breaks, the group's prior rating distribution, and anonymized one-line justifications from earlier rounds. Panelists accept or replace the action and say whether the agent remains at fault afterward. The fault judgment sets cancel-versus-overshadow per rule pair. Outliers are prompted for a justification in the usual Delphi way. Rounds stop when the weight posterior's decision fragility on a fixed probe set falls below a threshold, or after four rounds.

Baselines

The naive reference point is the hand-set weights a careful engineer would write, following the practice of Neufeld et al. and Hatschka et al.. Two independent engineers each set one version, and both are evaluated. Beating this is the minimum bar for the whole enterprise to matter.

The classical baseline is abstract-rating Delphi. Panelists rate each rule's severity and pairwise precedence on Likert scales across three rounds with median and interquartile feedback. Medians are then mapped to weights by a fixed monotone rescaling and precedence majorities to priority levels. This is what a competent Delphi researcher would do today, so it is the fairest test of whether consequence feedback adds anything. It receives the same panel size, the same total number of items answered per panelist, and the same wall-clock budget.

The strongest competitor is one-shot scenario fitting. A fresh panel answers the same total number of concrete scenario judgments in a single round, with scenarios drawn at random from the simulator. The weights are fit with the identical choice model. This isolates the two ingredients that distinguish the proposal, solver-chosen fragile scenarios and iterative feedback showing the implied action, from simply asking about concrete cases. An ablation splits these two apart: one arm uses solver-chosen scenarios but gives classical statistical feedback with no implied action shown, and another shows implied actions on randomly chosen scenarios.

A language-model arm asks a frontier model to set the weights and priority levels from the rule texts. The only related reference figure is GPT-4's roughly 80% on Commonsense Norm Bank, against Delphi's 92.8%, as reported in Investigating machine moral judgement through the Delphi experiment. That is a single-action judgment task, not a contrary-to-duty one. There is no reported result found on Lacuna for any method's agreement with expert panels on breach-repair choices, so the expert-agreement numbers here will be first measurements.

For enforcement, the reference is the existing supervisor. On Vegan Pac-Man it reaches 0.00 ghosts eaten and a 92% win rate, against 1.70 and 87% unshielded (Neufeld et al.), with ASP running about 50% faster than DDL at under 7 seconds per solve (Hatschka et al.). Elicited weights must preserve this level of compliance when the hidden norms are known.

Beating the strongest baseline means the following. Held-out agreement with an independent validation panel must be higher at equal judgment count by a margin exceeding the panel-resampling confidence interval. In addition, norm-erasure rate must be no worse.

Why not just…?

Why not just run a normal Delphi and map medians to weights? Severity ratings are elicited one rule at a time, while repair decisions depend on how rules combine in a specific breach. Nothing guarantees that a median of 6 for "notify operator" and 8 for "exit by shortest route" reproduces what the same experts would choose when exiting fastest crosses a privacy buffer. The abstract-rating Delphi arm tests this directly. If it matches the proposal on held-out agreement, the core claim is false and the contribution reduces to a compiler from Delphi output to weak constraints.

Why not just collect scenario choices once and fit weights? This is the real threat. It is the same estimator without iteration, and it may well capture most of the gain. The proposal's argument is that fragile scenarios carry far more information about decision-relevant weights than random ones. Random breach scenarios are mostly easy, with one obviously cheapest repair. Feedback should also let panelists correct misreadings that a single round bakes in. The one-shot arm and the two ablation arms settle this. If one-shot fitting matches at equal judgment count, the right conclusion is that experts should be surveyed on scenarios, not deliberated with. That would be a useful negative result about Delphi's value in this setting.

Why not just let a language model write the weights, or fold penalties into the reward? A model setting weights inherits whatever its training data says about drones and offers no accountable provenance. The inverted-label failure in Olson and Forbus is a warning about ungrounded norm sources, though not about this exact task. The language-model arm measures the gap rather than assuming it. Reward shaping is set aside for a different reason: it collapses a breach and its repair into one scalar. That loses the overshadowing distinction this proposal explicitly elicits. It also requires retraining whenever the panel's weights change, which the supervisor design of Neufeld et al. avoids.

Proposed Experiments

The first experiment is a synthetic recovery study run before any humans are involved. Simulated panelists each draw hidden weights around a population mean. Their abstract ratings are a noisy monotone distortion of those weights, and their scenario choices follow the choice model above. On Passive Vegan Pac-Man the population mean is set to the hand-built supervisor's norms. The check is whether the pipeline recovers that supervisor's decisions and its 0.00-ghosts behavior. On the drone world, the experiment measures weight recovery error and decision agreement as functions of the number of judgments for every arm. This validates the machinery, but it builds in the abstract-concrete gap by assumption, so it cannot confirm the core claim.

The decisive experiment is human. Four panels of about eighteen aviation experts each run, respectively, abstract Delphi, one-shot scenario fitting, solver-in-the-loop Delphi, and the feedback-only ablation. A fifth arm runs the scenario-only ablation if recruitment allows. A separate validation panel labels 150 held-out breach scenarios that no arm saw, with fault judgments included. The primary metric is the fraction of held-out scenarios where the supervisor's chosen action under each arm's weights matches the validation majority.

Secondary metrics are four. The first is norm-erasure rate: the fraction of simulated breaches after which the violated primary duty is absent from the supervisor's violation log while the validation panel says fault persists. The second is the number of inconsistencies surfaced and resolved during rounds, scored by $I_\wp$. The third is round-to-round interquartile shrinkage, the classical Delphi notion of consensus. The fourth is panelist time and attrition, since slowness and dropout are Delphi's documented weaknesses.

Finally, each arm's weights are deployed in the drone simulator over a shared set of breach-inducing episodes. Primary-duty compliance and correct triggering of repair duties are counted to confirm that better agreement does not cost enforcement quality.

The result that would most change the plan is solver-in-the-loop Delphi failing to beat one-shot scenario fitting at equal judgment count. In that case the defensible recommendation is the cheaper single-round survey, and the paper would report it as such.