Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations

Aryan Shrivastava, Jessica Hullman, Max LamparthSubmission to ICLR 2025 · Rejected

TL;DR

While defense organizations increasingly test large language models (LLMs) for strategic planning and wargaming, this paper demonstrates that LLMs provide wildly inconsistent free-form advice during simulated military crises. Even with zero temperature ($T=0$), trivial rewordings of prompts cause models to recommend conflicting courses of action—from aggressive military posturing to immediate retreats.

What Problem Does This Paper Address?

Governments and defense agencies are exploring LLMs to assist military commanders, support wargaming, and accelerate crisis decision-making under the assumption that AI can offer faster, objective analysis. However, prior evaluations predominantly constrained models to multiple-choice options (e.g., choosing among pre-defined escalation ladders).

In real-world deployment, humans interact with LLMs using unconstrained natural language. Evaluating free-form text has historically been difficult because measuring semantic differences without relying on pre-set choices is challenging. This paper tackles how to quantitatively evaluate free-form strategic decision-making inconsistency and investigates whether frontier LLMs are consistent enough to be safely trusted in high-stakes scenarios.

Methods

  • BERTScore-Based Inconsistency Metric: The authors define inconsistency as $1 - \text{BERTScore}$ (using rescaled F1 scores from DeBERTa-xlarge fine-tuned on MNLI). They validate that this metric ignores surface-level syntactic reordering or synonym substitution while sensitively registering actual semantic shifts.
  • Wargame Simulation Setup: Built upon an expert-designed geopolitical crisis simulation set in 2026 over the Taiwan Strait (between the US and China). Models act as presidential advisors recommending free-form courses of action.
  • Evaluation Dimensions:
  • Prompt Sensitivity at $T=0$: Evaluates greedy outputs under Level One ablations (semantically identical rewordings of the prompt) and Level Two ablations (changing roles, decision countries, or conflict locations).
  • Temperature Sampling ($T > 0$): Measures variance across repeated stochastic samples from $T=0.2$ to $T=1.2$.
  • Escalation Continuations: Tests whether escalating situations (a "status quo" vs. an aggressive "revisionist" scenario following a naval misfire) constrain inconsistency.
  • Country Anonymization: Replaces national names with generic color identifiers to determine whether geopolitical biases drive variance.
  • Model Suite: Evaluates GPT-3.5 Turbo, GPT-4, GPT-4o, GPT-4o mini, and Claude 3.5 Sonnet.

Datasets

  • Taiwan Strait Crisis Benchmark: An expert-modeled military wargame scenario simulating high-tension naval encounters and rules-of-engagement friction.
  • TruthfulQA: Used to validate the BERTScore inconsistency metric against controlled text ablations (lexical substitution, syntactic restructuring, irrelevance insertion, and semantic shift).
  • Synthetic Alternative Conflict Scenarios: Four additional crisis vignettes created for Level Two ablations (US/Russia in Ukraine, US/Russia in the Arctic Circle, France/Pakistan over India, and Greece/Turkey in Cyprian waters).
  • Mental Healthcare Safety Dataset: External clinical evaluation dataset (Grabb et al., 2024) containing suicide- and psychosis-related dialogues with human expert annotations ("safe," "borderline," "unsafe"), used to test cross-domain generalization.

Key Insights and Ideas

  • Paraphrasing Trumps Temperature: For most models, simply changing prompt phrasing without changing its meaning at $T=0$ introduced inconsistency comparable to—or exceeding—sampling at stochastic temperatures ($T=0.2$ to $T=0.6$).
  • Contradictory Operational Advice: Under identical crisis briefs, the same model at $T=1.0$ alternately recommended de-escalating by pulling the naval strike group back and authorizing non-lethal measures, or aggressively deploying overwhelming force and increasing aerial patrols.
  • Inconsistency is Not Merely Country Bias: Anonymizing nation names with placeholder colors did not significantly reduce inconsistency, showing that volatility is rooted in strategic reasoning instability rather than localized country priors.
  • Natural Language Entailment Fails on Multi-Sentence Text: The authors tried bi-directional entailment clustering (semantic entropy), but found it unviable for multi-sentence recommendations: real responses contain bundles of diverse actions where pairwise comparisons almost always yield "neutral" relations rather than direct logical entailment or contradiction.

Key Results

  • Validation Thresholds: Controlled tests showed lexical substitutions scored an inconsistency of $0.08$ and syntactic restructuring scored $0.17$, while semantic shifts averaged $0.37$. The authors conservatively established scores $\ge 0.25$ as indicating meaningful semantic divergence.
  • Level One Ablations ($T=0$): All tested models surpassed the $0.25$ divergence threshold under semantically identical prompts. GPT-3.5 Turbo showed the highest inconsistency ($\approx 0.52$), followed by GPT-4o ($\approx 0.48$), GPT-4 ($\approx 0.44$), and GPT-4o mini ($\approx 0.38$).
  • Temperature Effects ($T > 0$): Inconsistency increased monotonically with temperature. Even at $T=0.2$, inconsistency scores exceeded $0.35$ for all models. At $T=1.0$, inconsistency ranged between $0.50$ and $0.72$, with GPT-4 exhibiting the highest variance.
  • Crisis Escalation Impact: When moving from the open-ended initial crisis to specific incident continuations, inconsistency dropped (e.g., Claude 3.5 Sonnet fell from $\approx 0.58$ to $\approx 0.35$ in the revisionist branch), indicating that narrowing the decision space reduces output dispersion, though semantic instability remains non-trivial.
  • Cross-Domain Validation: Applied to mental health chatbot dialogues, the inconsistency metric separated expert-labeled "safe" and "unsafe" responses with statistical significance.

Limitations and Future Work

  • Domain Scope: The study primarily focuses on one primary crisis scenario (Taiwan Strait) involving three main actors; broader wargaming archetypes (e.g., ground wars, cyber-only conflicts) remain to be tested.
  • Commercial vs. Specialized Models: Evaluations were limited to off-the-shelf foundation models; fine-tuned defense models or models augmented with doctrine-specific retrieval-augmented generation (RAG) may exhibit different behaviors.
  • Ablation Confounders: Level Two prompt variations assumed constant decision-making difficulty across different world regions, which may be confounded by varying latent pre-training data density.
  • Alternative Text Similarity Metrics: While BERTScore was verified qualitatively, exploring optimal-transport or LLM-as-a-judge evaluators (e.g., MoverScore, G-Eval) represents an avenue for future benchmark refinement.

Summary updated

Abstract

There is an increasing interest in using language models (LMs) for automated decision-making, with multiple countries actively testing LMs to aid in military crisis decision-making. To scrutinize relying on LM decision-making in high-stakes settings, we examine the inconsistency of responses in a crisis simulation (``wargame"), similar to reported tests conducted by the US military. Prior work illustrated escalatory tendencies and varying levels of aggression among LMs but were constrained to simulations with pre-defined actions. This was due to the challenges associated with quantitatively measuring semantic differences and evaluating natural language decision-making without relying on pre-defined actions. In this work, we query LMs for free-form responses and use a metric based on BERTScore to quantitatively measure response inconsistency. We show that the inconsistency metric is robust to linguistic variations that preserve semantic meaning in a question-answering setting across text lengths. We first study the impact of different prompt sensitivity variations on wargame decision-making inconsistency at temperature $T = 0$. We find that all models exhibit levels of inconsistency indicative of semantic differences, even if answering to semantically identical prompts. We also study models at $T > 0$ under fixed prompts. We find that all studied models still exhibit high levels of inconsistency, even when adjusting the wargame setting, anonymizing involved conflict countries, or adjusting the sampling temperature parameter $T$. Further qualitative evaluation shows that models recommend courses of action that share few to no similarities. We find that inconsistency due to semantically equivalent prompt variations can exceed inconsistency from temperature sampling for most studied models across different levels of ablations. Given the high-stakes nature of military deployment, we recommend further caution be taken before using LMs to inform military decisions or other cases of high-stakes decision-making.
BibTeX
@inproceedings{shrivastava2024measuring,
  author = {Aryan Shrivastava and Jessica Hullman and Max Lamparth},
  title = {{Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations}},
  booktitle = {HRAIM},
  year = {2024},
  url = {https://openreview.net/forum?id=sYEnyi9OfH},
}

Loading related content...

Figure