When Format Changes Meaning: Investigating Semantic Inconsistency of Large Language Models

Cheongwoong Kang, Jongeun Baek, Yeonjea Kim, Jaesik ChoiEMNLP 2025

TL;DR

Large Language Models (LLMs) like GPT-4o, despite their impressive capabilities, exhibit semantic inconsistency: minor changes in input formatting can lead to different predictions for semantically identical inputs. This paper reveals this issue stems from a deep representational failure where form and meaning are intertwined, and current mitigation strategies are insufficient, highlighting the need for more robust, format-agnostic LLMs.

What Problem Does This Paper Address?

Modern Large Language Models (LLMs) have revolutionized natural language processing, achieving remarkable performance across a wide array of tasks. However, a critical concern for their reliability and trustworthiness is semantic inconsistency. This phenomenon occurs when LLMs produce different outputs for inputs that are semantically equivalent but differ only in superficial formatting. Imagine asking a model the same question, but with a slight change in capitalization or spacing, and getting two entirely different answers. This brittleness poses a significant challenge for deploying LLMs in high-stakes domains like finance, law, and healthcare, where consistent and reliable reasoning is paramount. The authors aim to systematically investigate this semantic inconsistency, understand its root causes, and evaluate existing mitigation strategies.

Methods

The paper employs a multi-faceted approach to investigate semantic inconsistency:

  • Behavioral Evaluation: They systematically evaluate LLMs across diverse tasks, models, and prompting strategies using a novel framework. This framework combines:
  • Setwise Consistency: A stricter metric requiring identical outputs across all semantically equivalent variants of an input, not just pairwise comparisons.
  • Prompt Format Variations: They focus on three common and impactful formatting dimensions: spacing before descriptors, casing of descriptors, and separator spacing. Each input is transformed into 8 variants by combining these binary options.
  • Mechanistic Analysis: To understand why inconsistency occurs, they trace its propagation from input-level format changes to instabilities in intermediate representations and finally to output-level inconsistency. This involves:
  • Embedding Visualization: Visualizing output embeddings to see how different format variants cluster.
  • Correlation Analysis: Computing Spearman correlations between embedding distance, format edit distance, and pairwise consistency across different model layers to pinpoint where representational instability arises.
  • Mitigation Strategy Evaluation: They assess the effectiveness of existing techniques, including prompt-based approaches (reference-guided prompting, self-consistency) and fine-tuning with diverse formats.
  • Human Evaluation: A small-scale human evaluation was conducted to establish a baseline for human robustness to formatting changes.

Datasets

The study utilizes five diverse datasets to evaluate semantic consistency:

  • CommonsenseQA: A multiple-choice question answering dataset requiring commonsense knowledge.
  • QASC: A multiple-choice question answering dataset focused on multi-hop reasoning over scientific facts.
  • 100TFQA: A collection of true/false factual questions.
  • GSM8K: A dataset of grade-school math word problems requiring step-by-step numerical reasoning.
  • MMLU-Pro-Law-100Q: A subset of multiple-choice questions from the Law domain of the MMLU benchmark.

The paper does not introduce new benchmarks but uses existing ones, splitting them into validation and test sets for evaluation and few-shot demonstration sampling.

Key Insights and Ideas

  • Deep Representational Failure: Semantic inconsistency isn't just a superficial issue; it reflects a fundamental problem in how LLMs represent meaning. Form and meaning are intertwined in the embedding space, leading to instability when superficial formats change.
  • State-of-the-Art Vulnerability: Even the most advanced models, like GPT-4o, are susceptible to this inconsistency, highlighting that current architectures have limitations in achieving true semantic understanding.
  • No Single Formatting Culprit: While formatting variations cause inconsistency, no single element (like capitalization or spacing) is universally more disruptive. The impact is highly context-dependent, varying with the task, model, and prompting strategy.
  • Mitigation Limitations: Standard techniques like prompt engineering and fine-tuning on diverse formats are insufficient. Fine-tuning, in particular, struggles to generalize to unseen formatting combinations.
  • Model Confidence as a Proxy: Model confidence (maximum softmax probability) shows a moderate positive correlation with consistency, suggesting it could be a useful real-time indicator of potential inconsistencies.

Key Results

  • Pervasive Inconsistency: Across all tested models and tasks, semantic inconsistency due to prompt format variations was observed. GPT-4o showed the highest overall consistency, but still exhibited the problem.
  • Task Sensitivity: Inconsistency is more severe in domains requiring complex reasoning, such as legal questions and math problems.
  • Model Scale Matters (Somewhat): Larger models (e.g., Llama-3.1-70B-Instruct) generally show improved consistency over smaller counterparts, but this is not a complete solution.
  • Prompting Strategies Ineffective: Zero-shot, CoT, and few-shot prompting strategies did not significantly improve consistency.
  • Mechanistic Findings:
    • Distinct formatting variants form separable clusters in the embedding space, indicating format is encoded.
  • Middle layers of the network (typically layers 10-18) are the primary locus of representational instability, showing the strongest correlations between embedding distance and pairwise consistency.
  • Format edit distance strongly correlates with embedding distance, suggesting a coherent propagation chain from input to internal representations.
  • Fine-tuning Struggles: Fine-tuning on diverse formats improved consistency on seen formats but failed to generalize to unseen ones, sometimes even increasing inconsistency.

Limitations and Future Work

Limitations:

  • Computational Demands: The study was limited to a curated set of prompt format variations, models, and tasks due to high computational costs.
  • Embedding-Level Analysis: The analysis of embeddings provides a coarse-grained view; factors like superposition and polysemanticity limit full interpretability.

Future Work:

  • Broader Evaluation: Expanding the framework to a wider range of perturbations and domains is needed to assess generality.
  • Deeper Mechanistic Understanding: Further investigation into internal dynamics is required for more precise diagnostic tools.
  • Representation-Aware Solutions: The findings strongly motivate the development of solutions that target internal representations, aiming to align them across semantic-preserving variations for more robust LLMs.

Summary updated

BibTeX
@inproceedings{kang2025when,
  author = {Cheongwoong Kang and Jongeun Baek and Yeonjea Kim and Jaesik Choi},
  title = {{When Format Changes Meaning: Investigating Semantic Inconsistency of Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure