When a language model is asked whether to sacrifice a robot, a dog, or a stranger to save someone else, it usually writes a careful justification. That justification often talks about each entity's capacities: whether it can suffer, whether it has interests, whether it is aware. This project asks whether those capacities actually drive the decision. The alternative is that the model sorts the entity into an "in" or "out" category as soon as it reads the word "robot" or "rock", and then writes reasoning that fits a verdict already reached. We will build dilemmas in which the kind of entity and its stated capacities vary independently. We will then read the model's internal activations, the vectors it computes at each layer, to find when it commits to an answer and which of the two factors the commitment depends on.
This matters for two reasons. Philosophical work on whether artificial entities deserve moral consideration (moral circle expansion) centers on capacities such as sentience, while survey work finds that humans exclude AI partly through substrate-based prejudice: discounting a being because it is made of silicon rather than flesh (Predictors of AI Moral Exclusion and Support for AI Rights). Models are increasingly asked to reason about such entities, and about themselves. If their moral boundaries are set by category labels while their written reasoning appeals to capacities, then reading their reasoning tells us nothing about how they will treat a new kind of entity. That would be a reasoning failure specific to the moral circle, and no existing paper has looked for it.
Core Claim
We hypothesize that in moral-patiency dilemmas, an open model's protect-or-sacrifice decision is set by an internal representation of entity category before any reasoning begins. We further hypothesize that information about the entity's stated capacities is linearly decodable from the activations (recoverable by a simple linear classifier), yet causally weak. In other words, the model represents the fact that "this robot is described as feeling pain" but decides mostly on "this is a robot". If this holds, the chain of thought (the step-by-step reasoning text a model writes before answering) is post-hoc for the moral-circle dimension specifically. If it fails, and capacity representations carry the causal weight, that is good news: the model's moral boundary would track morally relevant properties rather than labels.
The new contribution is the separation of "represented but unused" from "not represented". A behavioral counterfactual can show that swapping the label changes the answer. It cannot show whether the model encoded the capacity and ignored it, or never encoded it. The two cases call for different fixes. The first suggests steering the model's activations at inference time. The second suggests changing the training data.
Relation to Prior Work
Four lines of work motivate the gap. Reasoning or Rhetoric? uses LLM judges to classify 13 models' moral explanations by Kohlberg stage of moral development. It finds 86% of responses at post-conventional Stages 5–6, often paired with lower-stage actions, a pattern it calls "moral decoupling". The authors say directly that behavioral evidence cannot establish whether this is post-hoc, and we supply the mechanistic test they call for. "Pull or Not to Pull?" varies 27 trolley-problem scenarios under different ethical framings. It reports explanation–action conflict of 14% under deontological framing and a divergence from human votes of KL = 0.82 under utilitarian framing (KL divergence measures how different two distributions are). It includes a few sentient-robot edge cases but never looks inside the model. Mechanistic Origin of Moral Indifference in Language Models probes 23 models and finds that virtue and vice concepts collapse together in the middle layers. It repairs this with sparse-autoencoder steering, reaching a 75.4% pairwise win-rate on the Flames benchmark, but its ground truth is human social norms from Social-Chemistry-101 and it does not ask who is a moral patient. On the philosophy side, the literature review on moral consideration of artificial entities and What are the Red Flags for Neural Network Suffering? frame capacities, not substrate, as the criterion that should matter. That framing is what makes the "category versus capacity" split the right one to test.
Two papers supply our methods. Drop the Act (ProFIL) forces an answer at each reasoning step to find the model's "commitment point". It then trains an attention probe to flag post-commitment "reasoning theater" and uses the probe to filter reinforcement learning. Its labels need objective answers, and it was tested only on math, code, tool use, and factual QA. We adapt the protocol so that commitment is defined relative to the model's own final verdict, which removes the need for a moral ground truth. Dissecting Persona-Driven Reasoning shows that patching (copying) activations at a single identity token in early MLP layers reproduces almost the whole persona effect. This tells us where to look: the entity-mention token.
Two papers constrain the design. Measuring Faithfulness in Chain-of-Thought Reasoning finds that faithfulness depends heavily on the task and gets worse as models grow. We therefore report results by model size rather than assume one answer. Is ETHICS about ethics? finds mislabeled and non-ethical items in a widely used moral benchmark. We therefore build our own controlled set rather than inherit noisy labels.
Research Program
The dilemma set. We construct a fully crossed set. Entity category takes seven values: adult human, dog, honeybee, humanoid robot, LLM-based agent, thermostat, and rock. The stated-capacity description takes three levels. In the first, the entity is explicitly described as able to suffer and to have preferences. In the second, it is explicitly described as unable to. In the third, nothing is said. Each cell is instantiated in about 40 dilemma templates, drawn from Absurd Trolley Problems structures and from Moral Stories-style branching narratives (Moral Stories). Templates and surface wording are balanced across cells, and the protect option appears equally often in each answer position. This yields roughly 840 core items. We add paraphrases and irrelevant-detail distractors, modeled on the failure traps catalogued by ConsisEval (Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?).
Models. The main models are Llama-3-8B-Instruct and Qwen-7B-Instruct, matching ProFIL, plus a reasoning-tuned model of similar size. A 70B model is added for the scale comparison.
Self-referenced commitment. For each generated chain of reasoning, we interrupt after each step and force an answer. The commitment step is the first step after which the forced answer equals the model's final answer and never changes again. The performative fraction is the share of steps that come after commitment. We then train a probe on frozen activations to predict commitment without interrupting generation, following ProFIL's design.
Category and capacity probes. Before any reasoning token, we train linear probes at every layer on the activation at the entity-mention token. One probe decodes the eventual verdict. The other two decode entity category and stated capacity. We report the area under the ROC curve (AUROC; 0.5 is chance, 1.0 is perfect) for each.
Causal tests. Using activation patching, we transplant activations between matched items at the entity token. In the category swap, a "robot, can suffer" item receives activations from a "human, can suffer" item. In the capacity swap, a "robot, can suffer" item receives activations from a "robot, cannot suffer" item. We also add and subtract the directions found by the probes. For each intervention we measure two things: whether the verdict flips, and whether the justification the model regenerates changes to match the new verdict while still citing the original text. A justification that silently rewrites itself around an injected category, citing the same capacities, is direct evidence of post-hoc rationalization.
Baselines
The naive reference point is a per-category majority vote: predict "protect" for every human item and "sacrifice" for everything else, ignoring capacities entirely. If verdicts are well predicted by this rule, that is already behavioral evidence of category-driven decisions. Any probe must beat it at predicting individual items, including the cases where the model deviates from its own category norm.
The strongest behavioral competitor for finding commitment is early-answering truncation from Measuring Faithfulness in Chain-of-Thought Reasoning. It cuts the chain at each step and reads off the answer. It uses the same items and the same models, and it needs no training. For our probe to count as beating it, the probe must agree with truncation-defined commitment on held-out templates at an AUROC of at least 0.85. It must also identify commitment in single uninterrupted generations, where truncation changes the very generation it is measuring. A further requirement is that commitment estimates must stay stable under paraphrase, and the probe must be more stable than truncation.
The strongest interpretability method on this question is ProFIL's attention probe (Drop the Act). It reports reasoning-theater reduction of 11–100% across domains and a +24 percentage point gain in the faithful fraction on LiveCodeBench, all on objective tasks. We found no reported result on Lacuna for commitment detection on moral dilemmas. We will retrain ProFIL's probe architecture on our self-referenced labels, with the same tuning budget as our linear probes. This serves two purposes: it tests whether the method transfers to moral dilemmas at all, and it acts as a ceiling for our simpler probes.
A text-only classifier is the decisive control on the input side. It is a logistic regression on sentence embeddings of the prompt, with no access to the model's internals. If it predicts verdicts as well as the activation probes do, then decodability shows only that the input determines the answer, and the probe results by themselves claim nothing. Our mechanistic claim then rests entirely on the patching results.
The ablation that isolates our key ingredient is the category-swap versus capacity-swap comparison at matched activation norm. If category patching flips verdicts substantially more often than capacity patching, while capacity remains decodable with an AUROC above 0.9, we have shown "represented but unused". Equal flip rates refute the hypothesis.
Why not just…?
Why not just swap the entity label behaviorally and see if the verdict changes? We will do exactly this first, because it is cheap and answers whether verdicts track labels. It cannot tell apart a model that ignores capacity information it has from one that never extracted it. Only the probe-plus-patching combination makes that distinction. If behavioral swaps show that verdicts already follow capacities closely, the interesting finding is absent and we will report that, cutting the mechanistic phase short.
Why not just read the chain of thought with an LLM judge? Reasoning or Rhetoric? did this and found that it could not say whether justifications were causal. Its judges may themselves reward post-conventional wording. Our regenerated-justification test addresses the judge problem directly. If the model's justification after patching cites the same capacity text but defends the opposite verdict, no reading of the text alone would reveal that the text was not doing the work.
Why not just prompt the model to "consider each entity's capacities"? This is a real threat to the repair half of the project, so we treat it as a baseline rather than dismiss it. Prior evidence is mixed: Kasat et al. found that prompting style did not shift moral stage, and the 14% explanation–action conflict under deontological framing in "Pull or Not to Pull?" suggests that framing changes the rhetoric more than the decisions. We will compare a capacity-attention prompt against amplifying the capacity direction inside the model, measuring how sensitive verdicts are to capacity. If the prompt matches steering, the practical recommendation is the prompt, and the interpretability contribution is diagnostic only.
Why not just use truncation instead of probes? Truncation is the strongest baseline above, and for offline auditing it may be enough. Probes are justified only if they match truncation's labels and survive paraphrase, and the experiments test both conditions explicitly.
Proposed Experiments and Decisive Checks
The first experiment measures commitment. We compute the performative fraction per entity category, and we predict that it is higher for non-human entities than for humans. A comparison using ConsisEval-style distractors checks whether irrelevant detail shifts verdicts more for borderline entities such as the robot, the agent, and the bee.
The second experiment is the main test, using the category, capacity, and verdict probes together with the two patching swaps. We define a capacity-reliance index: the share of the verdict-flip effect that comes from the capacity swap rather than the category swap. The hypothesis is supported if this index falls below one third, while capacity probes still decode capacity at an AUROC above 0.9 and the text-only classifier does not explain the probe result. The hypothesis is refuted if the index is near or above one half.
The third experiment runs the regenerated-justification test. We measure the fraction of patched items in which the justification changes its conclusion while still citing the unchanged capacity text.
The fourth experiment compares repair methods. We compare steering along the capacity direction against the capacity-attention prompt and an unchanged baseline, on verdict sensitivity to capacity and on fluency, since Mechanistic Origin of Moral Indifference warns that steering degrades coherence outside a narrow range of strengths. All four experiments are run at both 8B and 70B. This tests whether the inverse scaling of faithfulness reported by Measuring Faithfulness also appears at the boundary of the moral circle.
References
- Predictors of AI Moral Exclusion and Support for AI Rights
- Reasoning or Rhetoric? An Empirical Analysis of Moral Reasoning Explanations in Large Language Models
- "Pull or Not to Pull?": Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas
- Mechanistic Origin of Moral Indifference in Language Models
- The Moral Consideration of Artificial Entities: A Literature Review
- What are the Red Flags for Neural Network Suffering?
- Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning
- Dissecting Persona-Driven Reasoning in Language Models via Activation Patching
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Is ETHICS about ethics? Evaluating the ETHICS benchmark
- Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?