TL;DR
This paper tackles the "fanaticism problem" in moral uncertainty: how to aggregate multiple ethical theories when one theory, even with low confidence, can dominate decision-making. The authors formalize fanaticism and propose "k-trimmed Highest Mean" and "Highest Median" as less fanatical alternatives to the standard "Maximizing Expected Choiceworthiness" (MEC) approach.
What Problem Does This Paper Address?
When an agent (human or AI) faces ethical dilemmas, there's often no single, universally agreed-upon ethical theory. The field of "moral uncertainty" proposes that agents should consider multiple ethical theories, each with a certain degree of confidence (credence). The challenge is how to combine these theories to make a decision. A prominent method, Maximizing Expected Choiceworthiness (MEC), aggregates theories by weighting their evaluations by their credences. However, MEC suffers from "fanaticism": a theory with very low credence can still dictate the agent's actions if it assigns extremely high or low values to certain outcomes. This undermines the idea of democratically incorporating diverse ethical perspectives and can lead to undesirable behavior. The paper aims to formally define this fanaticism problem and offer solutions.
Methods
The authors frame moral uncertainty as a social welfare aggregation problem, where ethical theories are treated as "individuals" and their credences as "weights." They then analyze existing and propose new social welfare functionals (SWFs) that represent different aggregation methods.
- Formalizing Fanaticism: They define fanaticism not as a binary state but as a graded property of SWFs. A key concept is a "dominant subset" of theories: a group of theories that, regardless of other theories, dictates the final ordering of actions. Fanaticism occurs when a SWF is systematically vulnerable to such dominant subsets, especially those with low total credence.
- Proposed Alternatives:
- k-trimmed Highest Mean: This method modifies MEC by "trimming" the most extreme evaluations from a certain percentage (k) of theories, making it more robust to outliers (and thus, potentially, fanaticism).
- Highest Median: This is presented as an extreme form of trimming, focusing on the median evaluation, which is inherently less sensitive to extreme values.
Datasets
Not discussed in the paper. The paper uses a running example (FROBO the firefighter robot) to illustrate concepts rather than relying on empirical datasets.
Key Insights and Ideas
- Fanaticism is a Graded Property: The paper moves beyond a simple "fanatical" or "not fanatical" label, providing a formal, graded definition that allows for nuanced analysis of SWFs.
- MEC is "Pascalian" Fanatical: The authors prove that MEC is maximally vulnerable to fanaticism, meaning it's susceptible even with extremely low credences. This is a strong critique of its foundational role.
- Trimming as a Solution: The k-trimmed Highest Mean and Highest Median approaches demonstrate that by reducing the influence of extreme evaluations (especially from low-credence theories), fanaticism can be mitigated.
- Highest Median is Completely Non-Fanatical: This is a significant finding, suggesting it's a robust solution for aggregating ethical theories without succumbing to fanaticism.
Key Results
- MEC and Maximin are Pascalian Fanatical: The paper formally proves that both Maximizing Expected Choiceworthiness (MEC) and Maximin are susceptible to extreme fanaticism.
- k-trimmed Highest Mean is Partially Non-Fanatical: This method avoids fanaticism for low credence thresholds (k' < k) but can still be fanatical for higher thresholds.
- Highest Median is Completely Non-Fanatical: The Highest Median functional is proven to be not k-fanatical for any k, making it a strong candidate for resolving moral uncertainty.
Limitations and Future Work
- Choosing 'k' for k-trimmed Highest Mean: The paper acknowledges that determining the optimal value of 'k' for the k-trimmed Highest Mean remains an open question.
- Stakes Sensitivity vs. Credence Sensitivity: The authors note a tension between how sensitive an SWF is to the magnitude of ethical evaluations (stakes) versus how sensitive it is to the confidence in those evaluations (credences). Finding an appropriate balance is a key area for future work, potentially through a compromise like k-trimmed Highest Median with a moderate 'k'.
- Application to AI Agents: While the paper's results are applicable to AI agents learning ethical preferences, further exploration of how these formalisms can be integrated into practical AI systems is suggested.
Summary updated