Isolating Symbolic Reasoning via Semantic Obfuscation

This direction is merged into another direction: semantic obfuscation. Its last version stays readable and citable here.

Version 1 of 125 papers26 conceptsfrozen 2026-10-04

When a Large Language Model (LLM) successfully generates a plan to "stack Block A on Block B," is it actually reasoning through the physics and logic of the task, or is it simply recalling a similar sequence from its massive training corpus? This question sits at the heart of a growing body of research that seeks to distinguish between approximate retrieval—mimicking patterns—and principled reasoning—following abstract rules.

To solve this, researchers have converged on a powerful diagnostic tool: semantic obfuscation. By stripping away meaningful words and replacing them with arbitrary symbols, they force models to rely on pure logic rather than linguistic intuition.

The Illusion of Planning

On the surface, LLMs appear to be competent planners. They can suggest travel itineraries or write code for common algorithms. However, Subbarao Kambhampati argues that this is often an illusion created by the models' ability to perform "universal approximate retrieval." Because the internet is full of examples of "Blocks World" problems and standard programming tasks, the models can "look up" a plausible-looking answer without understanding the underlying constraints.

In Large Language Models Still Can't Plan, Karthik Valmeekam and colleagues demonstrated that even state-of-the-art models like GPT-4 struggle with basic autonomous planning. Their performance is not just low; it is brittle. When the names of the actions and objects are changed, the models' ability to solve the problem often vanishes.

The Mechanism: Semantic Obfuscation

Semantic obfuscation (also referred to as lexical perturbation or semantic masking) is the process of decoupling a problem's logical structure from its natural language labels.

Consider a standard planning task:

Action: Drive(truck, city_a, city_b) Precondition: At(truck, city_a)

In an obfuscated version, this becomes:

Action: Alpha(Object_1, Location_X, Location_Y) Precondition: State_P(Object_1, Location_X)

By replacing "drive" with "alpha" and "truck" with "Object_1," researchers like Matthew Marquez and Sarath Sreedharan isolate the model's ability to follow formal rules. If a model can solve the first version but fails the second, it is likely relying on common-sense priors—pre-trained associations about what "driving" usually entails—rather than a deep grasp of the logical preconditions and effects.

Why Obfuscation Reveals the "Reasoning Gap"

The reason obfuscation is so effective is that it disrupts the model's "System 1" processing—the fast, intuitive pattern matching that LLMs excel at. Subbarao Kambhampati describes LLMs as sophisticated System 1 machines that lack the "System 2" (slow, deliberative, logical) capabilities required for reliable planning.

This "reasoning gap" appears across various domains:

  • Programming: In LLMs Lean on Priors, Not Programming Language Semantics, Aditya Thimmaiah and Milos Gligoric show that models fail to execute code correctly if standard operators are redefined (e.g., making the + symbol perform subtraction).
  • Logic Problems: Tharindu Madusanka found that while Transformers can solve numerical satisfiability problems in-distribution, replacing nouns with abstract symbols reveals a reliance on superficial patterns rather than mathematical rules.
  • Tool Usage: Cheng-Yu Hsieh used obfuscated command sets to prove that models often struggle to use new tools unless they can map them back to known linguistic concepts.

The Evolution of Frontier Models

Recent "reasoning models" like OpenAI’s o1 (Strawberry) have attempted to bridge this gap using internal chain-of-thought processing. Karthik Valmeekam and Kaya Stechly evaluated these models in Planning in Strawberry Fields, finding a significant leap in performance.

However, even these models are not "master planners." André Grahl Pereira and Jendrik Seipp noted in their 2025 evaluation that while models like Gemini 2.5 Pro show remarkable improvements in purely symbolic reasoning, most frontier models still exhibit a performance drop when semantic hints are removed. They remain highly sensitive to the "naturalness" of the problem description.

From Idea Generators to Verifiable Planners

If LLMs struggle with autonomous, abstract reasoning, how should we use them? The research suggests a shift toward neuro-symbolic architectures.

Instead of asking an LLM to generate a final plan, researchers like Michael Katz and Harsha Kokel suggest using them as formalizers—translating natural language into symbolic languages like PDDL (Planning Domain Definition Language). These symbolic representations can then be verified by deterministic, classical planners.

As Kaya Stechly demonstrates, LLMs are poor at "self-correcting" their own logic, but they excel as "idea generators" when paired with an external symbolic verifier that provides structured, logical feedback. This "LRM-Modulo" framework uses the LLM's linguistic strengths to navigate the search space while relying on symbolic logic to guarantee the plan's correctness.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_isolating_symbolic_reasoning_via_semanti_2463f687,
  title = {Isolating Symbolic Reasoning via Semantic Obfuscation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/isolating-symbolic-reasoning-via-semantic-obfuscation/rdr_2463f68703d34793b3a44845e67479b0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_isolating_symbolic_reasoning_via_semanti_2463f687_v1,
  title = {Isolating Symbolic Reasoning via Semantic Obfuscation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/isolating-symbolic-reasoning-via-semantic-obfuscation/rdr_2463f68703d34793b3a44845e67479b0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}