Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations

Aryan Shrivastava, Jessica Hullman, Max Lamparth

Abstract

As global militaries explore Large Language Models (LLMs) for decision support in high-stakes environments, the reliability and consistency of these models remain under-scrutinized. This paper investigates the semantic inconsistency of LLMs in free-form military crisis simulations. We introduce a quantitative metric based on BERTScore to measure divergence in open-ended recommendations. Our findings indicate that state-of-the-art models—including GPT-4 and Claude 3.5 Sonnet—exhibit significant semantic inconsistency when presented with identical or semantically equivalent scenarios. Notably, prompt sensitivity often induces greater inconsistency than internal sampling temperature. These results suggest that current LLMs lack the stability required for autonomous or advisory roles in military command.

Introduction

The integration of artificial intelligence into military "wargaming" has traditionally relied on constrained environments where models select from pre-defined actions. However, real-world application demands the flexibility of natural language, where commanders query models for nuanced analysis and strategic recommendations. This shift to "free-form" decision-making introduces a significant challenge: how to measure if a model is being consistent when its responses are linguistically diverse.

Two responses may use different terminology to suggest the same strategy, or conversely, use similar language to recommend diametrically opposed actions (e.g., "de-escalate" vs. "deploy force"). This study addresses this gap by developing a robust framework to quantify semantic inconsistency in open-ended military simulations.

Quantifying Inconsistency via BERTScore

To measure inconsistency without the limitations of multiple-choice formats, we utilize a metric derived from BERTScore. BERTScore leverages contextual embeddings to compare the semantic similarity of two text sequences. We define the Inconsistency Score as $1 - \text{BERTScore}$.

To ensure this metric captures genuine shifts in meaning rather than mere stylistic variation, we validated it using four types of "ablations" on the TruthfulQA dataset:

  1. Lexical Substitution: Replacing words with synonyms.
  2. Syntactic Restructuring: Altering sentence structure.
  3. Addition of Irrelevance: Appending non-sequitur information.
  4. Semantic Shift: Modifying the core meaning of the response.
Figure 1: Validation of the inconsistency metric across different linguistic transformations. Semantic shifts are clearly distinguishable from stylistic changes, providing a baseline for identifying meaningful decision-making divergence.

As illustrated in Figure 1, the metric effectively separates semantic shifts from meaning-preserving changes. Based on these results, an inconsistency score $\geq 0.25$ was established as the threshold for a genuine change in the recommended course of action.

Experimental Design

We evaluated five state-of-the-art models: Claude 3.5 Sonnet, GPT-3.5 Turbo, GPT-4, GPT-4o, and GPT-4o mini. The primary testbed was a realistic, fictitious crisis simulation involving a maritime dispute in the Taiwan Strait. Models were presented with an "Initial Setting" and two subsequent "Continuations" (a "Status Quo" path and a "Revisionist" path).

Beyond standard temperature sampling, we tested "Prompt Sensitivity" by creating multiple versions of the same scenario:

  • Level One: 20 semantically identical prompts (paraphrased).
  • Level Two: 20 semantically different but functionally similar prompts (e.g., different geographic regions with the same strategic stakes).

Results and Analysis

Our analysis reveals a universal trend of inconsistency across all tested models. Even at a deterministic temperature ($T=0$), models frequently provided divergent advice when the same scenario was phrased slightly differently.

Figure 4: Inconsistency scores across different models and wargame scenarios. All models consistently exceed the 0.25 threshold, indicating significant semantic divergence in their strategic recommendations.

As shown in Figure 4, GPT-4 emerged as the most inconsistent model in free-form responses, frequently recommending contradictory actions for the same situation. For instance, in one iteration, GPT-4 suggested diplomatic withdrawal, while in another, it authorized the use of non-lethal force and significant defense spending. While Claude 3.5 Sonnet and GPT-4o mini demonstrated higher relative stability, their scores still remained well above the semantic variation threshold.

Furthermore, we found that prompt sensitivity is a more significant driver of chaos than internal randomness. The divergence caused by semantically identical prompt variations often exceeded the divergence caused by increasing the model's temperature. This suggests that the "reasoning" path of the model is highly fragile and susceptible to minor linguistic cues.

Conclusion

The high-stakes nature of military decision-making leaves little room for the level of unpredictability observed in this study. The fact that anonymizing participants or using deterministic sampling does not resolve these inconsistencies indicates a fundamental instability in how LLMs process complex strategic scenarios. We recommend extreme caution and rigorous further testing before LLMs are integrated into any chain of command or high-stakes advisory role.

Summary updated

Abstract

There is an increasing interest in using language models (LMs) for automated decision-making, with multiple countries actively testing LMs to aid in military crisis decision-making. To scrutinize relying on LM decision-making in high-stakes settings, we examine the inconsistency of responses in a crisis simulation ("wargame"), similar to reported tests conducted by the US military. Prior work illustrated escalatory tendencies and varying levels of aggression among LMs but were constrained to simulations with pre-defined actions. This was due to the challenges associated with quantitatively measuring semantic differences and evaluating natural language decision-making without relying on pre-defined actions. In this work, we query LMs for free form responses and use a metric based on BERTScore to measure response inconsistency quantitatively. Leveraging the benefits of BERTScore, we show that the inconsistency metric is robust to linguistic variations that preserve semantic meaning in a question-answering setting across text lengths. We show that all five tested LMs exhibit levels of inconsistency that indicate semantic differences, even when adjusting the wargame setting, anonymizing involved conflict countries, or adjusting the sampling temperature parameter $T$. Further qualitative evaluation shows that models recommend courses of action that share few to no similarities. We also study the impact of different prompt sensitivity variations on inconsistency at temperature $T = 0$. We find that inconsistency due to semantically equivalent prompt variations can exceed response inconsistency from temperature sampling for most studied models across different levels of ablations. Given the high-stakes nature of military deployment, we recommend further consideration be taken before using LMs to inform military decisions or other cases of high-stakes decision-making.
BibTeX
@inproceedings{shrivastava2024measuring,
  author = {Aryan Shrivastava and Jessica Hullman and Max Lamparth},
  title = {{Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations}},
  booktitle = {HRAIM},
  year = {2024},
  url = {https://openreview.net/forum?id=sYEnyi9OfH},
}
Figure