When a human or a traditional computer program evaluates a logical argument, they rely on symbolic invariance. If the premise states that All X are Y, and all Y are Z, the conclusion All X are Z is self-evident. It holds true whether $X, Y, Z$ represent concrete concepts like "apples, fruits, food" or abstract variables like "$\alpha, \beta, \gamma$". The underlying logic is entirely decoupled from the surface-level tokens.
Large Language Models (LLMs) break this assumption. A rapidly growing body of research reveals a phenomenon known as semantic dependency: a model's ability to execute a logical chain is tightly tethered to the familiarity and contextual meaning of the words used to describe it. When these familiar "semantic cues" are removed or obfuscated, the model's reasoning capacity frequently collapses, revealing that what appears to be abstract logic is often a sophisticated form of pattern retrieval.
The Mechanism: Approximate Retrieval vs. Deliberative Execution
To understand why semantic dependency occurs, we must look at how language models process reasoning tasks. LLMs are trained to predict the next token based on statistical co-occurrences in massive text corpora. In these corpora, logical reasoning is almost always embedded within meaningful real-world contexts.
Consequently, when an LLM is asked to solve a planning task or a logic puzzle, it does not compile an internal, domain-independent logical execution engine. Instead, it performs what Subbarao Kambhampati terms universal approximate retrieval in the single-author paper Can Large Language Models Reason and Plan?. The model leverages "System 1" intuitive pattern matching to recall token transitions associated with common-sense actions (e.g., "open the door," "fetch the blocks").
These familiar terms act as lexical anchors. When the problem description uses standard terminology, the model's attention mechanisms easily map the input to high-probability pathways derived from its training data. However, if those anchors are severed, the statistical retrieval mechanism fails. The model cannot fall back on pure "System 2" deliberative reasoning because it lacks a mechanism to execute abstract rules independently of linguistic context.
The Evidence from Semantic Obfuscation
Researchers expose this dependency through semantic obfuscation—experiments that alter the surface-level names of actions, objects, or labels while keeping the underlying logical or mathematical constraints exactly the same.
1. Planning and Action Sequences
In classical planning benchmarks like PlanBench, models are evaluated on their ability to generate valid sequences of actions to achieve a goal. Karthik Valmeekam et al. demonstrated the fragility of this capability in Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change). When the authors replaced common-sense action labels with abstract or misleading terms, model performance collapsed.
Even when evaluating state-of-the-art "Large Reasoning Models" designed for multi-step thinking, Karthik Valmeekam et al. found in LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench that semantic obfuscation remains a significant bottleneck, causing severe performance drops when action names and predicates are replaced with randomized strings.
2. Logic Puzzles and Symbol Remapping
A similar pattern emerges in deductive reasoning. Henrike Beyer et al. introduced the Mystery-Zebra benchmark in Lexical Recall or Logical Reasoning: Probing the Limits of Reasoning Abilities in Large Language Models. They discovered that while frontier models excel at famous logic puzzles when presented with standard terminology, their performance drops precipitously toward random baselines when the same logical constraints are hidden behind unfamiliar words.
This "abstraction gap" is formalized by Qingchuan Ma et al. in Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective, where systematic symbol remapping proves that current models rely heavily on memorized token patterns rather than abstract rule comprehension.
3. Task Labels and In-Context Learning
The dependency extends down to basic classification tasks. In Do Prompt-Based Models Really Understand the Meaning of Their Prompts?, Albert Webson et al. showed a significant performance gap when models are forced to process arbitrary or reversed task labels. The models struggle to decouple the underlying task logic from fixed, pre-trained lexical mappings.
Standard Logic (High Accuracy)
[Premise: Alice is taller than Bob] ──> (Triggers familiar spatial pathways) ──> [Output: Alice is taller]
Obfuscated Logic (Low Accuracy)
[Premise: Qx9 is glimpish than Zp2] ──> (No familiar pathways triggered) ──> [Output: Failure / Random]
The Language-Centric Paradox
Because language models are optimized for linguistic patterns, stripping away natural language and forcing them to reason directly on raw symbols actually harms their performance. This creates a paradox: while a human logician prefers abstract symbols to avoid real-world biases, an LLM requires natural language "glue" to activate its internal reasoning pathways.
For instance, Yebowen Hu demonstrates in When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives that when tracking and aggregating data points across a sequence, replacing descriptive names with abstract symbolic identifiers severely degrades a model's tracking capacity. The model relies on natural language semantic patterns to organize and maintain state.
To exploit this quirk, Yile Wang et al. developed the Symbol-to-Language (S2L) method in Speak It Out: Solving Symbol-Related Problems with Symbol-to-Language Conversion for Language Models. Instead of forcing the model to compute directly with raw symbols, S2L translates abstract formulas and delimiters into descriptive text. By re-introducing semantic context, the authors significantly boosted the reasoning performance of models like GPT-4, proving that language-centric models require linguistic scaffolding to simulate abstract logic.
Overcoming the Semantic Bottleneck
To move past the limitations of semantic dependency, researchers are exploring two primary paradigms: post-training interventions and neuro-symbolic architectures.
Post-Training Abstraction
One approach is to explicitly train models to ignore spurious lexical correlations. In Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training, Shuiwang Ji et al. introduced a post-training framework (CAPT) that uses symbolic abstraction and causal interventions. By forcing models to solve tasks where real-world associations are intentionally inverted or replaced with nonsense during training, they decouple the reasoning process from specific entity labels, resulting in much higher out-of-distribution robustness.
Neuro-Symbolic Verification
Rather than trying to force an autoregressive language model to act as a flawless logical engine, a parallel track of research advocates for dividing the labor. In Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks, Subbarao Kambhampati et al. propose the LLM-Modulo Framework.
In this paradigm, the LLM is not trusted to execute the logic. Instead, the model acts as a flexible, creative "System 1" generator that translates natural language problems into structured, formal representations. These representations are then passed to an external, mathematically rigorous "System 2" symbolic verifier (such as a classical planner or theorem prover). This neuro-symbolic combination bypasses the semantic dependency bottleneck entirely, leveraging the linguistic strengths of LLMs while guaranteeing logical correctness through external code.