Decoupled Perception and Reasoning Architectures

306 concepts225 papers1 proposal

When modern Multimodal Large Language Models (MLLMs) fail, they often do so in puzzling ways. A model might perfectly describe every object in an image, yet completely fail a basic logic puzzle about how those objects interact. Conversely, it might confidently hallucinate a fact because its strong linguistic priors override the actual pixels it sees.

This vulnerability stems from a fundamental architectural tension: monolithic models attempt to map raw sensory data directly to high-level logical reasoning within a single, end-to-end network. To resolve this, a prominent direction in AI research focuses on decoupling perception from reasoning. By separating the "eyes" (sensory extraction) from the "brain" (logical inference), modular architectures allow each component to excel in its native domain.

The Core Tension: Visual Distraction vs. Linguistic Priors

End-to-end multimodal networks frequently suffer from interference between their visual and textual processing layers. As Jingqi Zhou et al. observe in ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom, monolithic models often rely too heavily on internal linguistic knowledge ("wisdom") while ignoring the actual visual tokens ("eyesight"). This leads to answers that sound plausible but are factually ungrounded.

At the same time, the inverse problem occurs: raw visual inputs can overwhelm a model's logical consistency. Jiwan Chung et al. demonstrate in VLIS: Unimodal Language Models Guide Multimodal Language Generation that integrated architectures are prone to "visual distraction," where complex image contexts inadvertently override basic commonsense reasoning.

Decoupled architectures eliminate this interference by introducing a strict boundary. Instead of passing raw visual embeddings directly into a shared attention space, the perception module must first translate its findings into a discrete, symbolic format—such as text, tables, or coordinates—before the reasoning module ever processes it.

Modality Translation via Symbolic Intermediates

The simplest way to implement this decoupling is through a two-stage translation pipeline. The goal is to convert specialized sensory data into natural language or structured text, transforming a multimodal problem into a pure text-reasoning task.

[Raw Image/Audio] ──> [Perception Module] ──> [Symbolic Text/Table] ──> [Text-Only LLM] ──> [Final Answer]

This approach is highly effective for structured visual data. In DePlot: One-shot visual language reasoning by plot-to-table translation, Fangyu Liu et al. decouple chart reasoning by using a specialized image-to-text module that translates plots into linearized tables. A standard text-only LLM then performs the necessary mathematical and logical inference on the table. Because the reasoning engine does not need to learn how to parse pixels, it can focus its full parameter capacity on computation and logic.

This separation also reveals exactly where multimodal systems break down. Single-author researcher Xuejing Liu demonstrates in What Large Language Models Bring to Text-rich VQA? that a modular pipeline combining a standalone Optical Character Recognition (OCR) tool with a text-only LLM frequently outperforms massive end-to-end multimodal models. This indicates that the primary bottleneck in text-rich visual tasks is often the precision of the visual extraction itself, rather than the model's core capacity to reason.

Similarly, abstract reasoning benefits from this translation step. Giacomo Camposampiero et al. show in Abstract Visual Reasoning Enabled by Language that complex visual puzzles can be solved effectively by converting them into text descriptions, thereby unlocking the implicit logical priors embedded in text-pretrained models.

Agentic Orchestration and Iterative Feedback

While a one-way pipeline works well for straightforward translation, complex tasks require a dynamic interplay between seeing and thinking. Modern decoupled architectures achieve this by establishing an agentic information flow, where the text-based reasoning engine acts as a controller that actively queries specialized perception tools.

                  ┌─────── Poses Sub-Questions ──────┐
                  ▼                                  │
          [Reasoning Engine]                  [Perception Tool]
                  │                                  ▲
                  └────── Processes Answers ─────────┘

In SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs, Zijia Liu et al. pair a high-capacity text model with a lightweight vision-language model acting as a "Translator Agent." Instead of relying on a single static caption, the text model guides the vision agent through a multi-round feedback loop, asking targeted questions about specific regions of the image as its logical chain unfolds.

This divide-and-conquer strategy is formalized by Haoxuan You et al. in IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models. The authors use a central LLM to break down a complex query into smaller, verifiable sub-questions, which are then answered sequentially by a vision module. By decomposing the task, the system prevents the fragile multi-step logic failures that frequently plague monolithic models when they attempt to process perception and reasoning simultaneously.

Why Decoupling Works: Independent Scaling and Bias Mitigation

The success of decoupled architectures comes down to two main factors: how neural networks scale, and how they handle bias.

1. Divergent Scaling Laws

Perception and reasoning are fundamentally different cognitive capabilities that develop best on different data distributions. Junlin Han et al. explore this in Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training, showing that visual priors in language models consist of distinct perception and reasoning components that follow entirely separate scaling laws. While object perception emerges from broad, diverse web corpora, reasoning capabilities are primarily driven by highly structured data like code and mathematics. Forcing both modalities into a single end-to-end representation can dilute the optimization of both. Decoupling allows researchers to scale the reasoning engine independently without retraining the visual front-end.

2. Breaking Superficial Shortcuts

Monolithic models often exploit statistical shortcuts in training data—for example, answering "What color is the banana?" with "yellow" without fully verifying the image. Decoupling forces the model to be honest. As Chenchen Jing et al. demonstrate in Overcoming Language Priors in VQA via Decomposed Linguistic Representations, splitting the reasoning process into an object referring stage (localizing the subject) and a verification stage (validating its attributes) prevents the model from relying on language shortcuts. The system is forced to find explicit visual evidence because the reasoning engine cannot see the raw pixels and must rely entirely on what the perception module explicitly extracts.

Ultimately, decoupling transforms a messy multimodal optimization problem into a clean, symbolic interface. By allowing specialized vision tools to handle perception and high-capacity language models to handle logic, these architectures provide a more transparent, efficient, and reliable path toward advanced visual reasoning.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoupled_perception_reasoning_architect_36ca89d5,
  title = {decoupled perception-reasoning architecture (decoupled perception and reasoning)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupled-perception-reasoning-architecture-decoupled-perception-and-reasoning/rdr_36ca89d5821e47d1ac3d5e969e8023a7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoupled_perception_reasoning_architect_36ca89d5_v1,
  title = {decoupled perception-reasoning architecture (decoupled perception and reasoning)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupled-perception-reasoning-architecture-decoupled-perception-and-reasoning/rdr_36ca89d5821e47d1ac3d5e969e8023a7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake