Disentangling and Integrating Relational and Sensory Information in Transformer Architectures

Awni Altabaa, John Lafferty

TL;DR

The Dual Attention Transformer (DAT) introduces a novel architectural extension that explicitly separates "sensory" information (what an object is) from "relational" information (how objects relate to one another). By integrating specialized relational attention heads alongside standard ones, DAT achieves significantly better data and parameter efficiency across tasks ranging from synthetic visual reasoning to large-scale language modeling.

What Problem Does This Paper Address?

While Transformers have revolutionized AI, they possess a known Achilles' heel: abstract relational reasoning. Standard self-attention is excellent at routing information based on similarity, but it primarily moves "sensory" features—the raw attributes of individual tokens or image patches. The relationships between these objects are often entangled with their features, making it difficult for the model to learn pure, abstract rules (e.g., "is object A the same shape as object B, regardless of color?").

Humans excel at this kind of out-of-distribution generalization because we can reason about structure independently of appearance. The authors argue that neural networks struggle here because they lack an explicit computational mechanism for encoding and processing these relationships. DAT aims to bridge this gap by providing a dedicated pathway for relational data.

Methods

The core innovation is the Dual Attention module, which runs two types of attention heads in parallel:

  1. Sensory Attention (Standard): The familiar mechanism where a query retrieves a "value" vector (sensory features) from attended objects.
  2. Relational Attention (Novel): Instead of retrieving features, this head retrieves a relation vector $r(x, y_i)$—a series of learned comparisons between the target and the source—and a symbol $s_i$.
Figure 1. Standard self-attention retrieves sensory information vi about the attributes of individual objects while relational attention retrieves relational information r(x, yi) about the relationship between the objects in the context and the target. Each relation is tagged with a symbol si which acts as an abstract variable identifying the source. In both cases, information is aggregated according to the attention scores αi, which are computed by a softmax over inner products of queries and keys.

Key Components:

  • Relation Vectors: These are computed as inner products across different learned feature subspaces, representing fine-grained comparisons.
  • Symbol Assignment: To process relations, the model needs to know which object a relation belongs to without necessarily knowing all its features. The authors use "symbols" as abstract pointers. They explore three mechanisms: Positional (index-based), Relative Position, and Symbolic Attention (grouping objects into learned equivalence classes).
  • Integration: The outputs of both head types are concatenated and passed through a standard MLP, allowing the model to integrate sensory and relational insights at every layer.

Datasets

The authors evaluated DAT across a broad spectrum of challenges:

  • Relational Games: A synthetic visual benchmark requiring models to identify relationships like "betweenness" or "matching patterns" in grids of geometric shapes.
  • Mathematics Benchmark: A suite of symbolic tasks including algebra, calculus (differentiation), and polynomial expansion.
  • CIFAR-10 & CIFAR-100: Standard image recognition tasks to test the Vision Transformer (ViT) variant, ViDAT.
  • FineWeb-Edu: A high-quality 10-billion-token dataset used to train decoder-only language models up to 1.3 billion parameters.

Key Insights and Ideas

  • Disentanglement is Power: By separating the "what" from the "how they relate," the model doesn't have to "re-learn" relationships every time a sensory feature (like color) changes. This leads to much faster convergence.
  • Hierarchical Reasoning: Because DAT layers are stacked, the model can perform higher-order reasoning. A second layer of relational attention can compute "relations between relations," which is essential for complex tasks like the "match pattern" game.
  • Semantic Pointers: In language models, visualizations suggest that relational heads learn to attend to semantic connections (e.g., connecting "model" to "mathematical" or "machine") rather than just syntactic ones like punctuation or proximity.
  • Symmetry as a Bias: For visual tasks, enforcing symmetry in the relation function ($r(x,y) = r(y,x)$) acts as a powerful inductive bias that further improves learning speed.

Key Results

DAT consistently outperformed standard Transformers in efficiency and accuracy:

  • Relational Games: DAT was vastly more sample-efficient. On the difficult "match pattern" task, DAT achieved high accuracy with a fraction of the data needed by a standard Transformer.
Figure 2. Learning curves on the relational games benchmark. Each subplot corresponds to a different task. Numbers in square brackets in legend labels indicate parameter counts. Solid lines indicate the mean over 5 trials with different random seeds and the shaded regions indicate bootstrap 95% confidence intervals. DAT is more data-efficient at relational learning compared to a Transformer.
  • Mathematics: DAT models outperformed standard Transformers across all tested scales (750K to 1.75M parameters) and tasks, particularly in algebra and polynomial math.
Math performance across model sizes
  • Language Modeling: At the 1.3B parameter scale, DAT achieved lower perplexity than standard Transformers while using fewer parameters, demonstrating that relational biases scale effectively to massive real-world datasets.
  • Vision: ViDAT achieved ~90% accuracy on CIFAR-10 with 6.0M parameters, beating a standard ViT that had 7.1M parameters and only reached ~86%.

Limitations and Future Work

  • Hardware Optimization: The current implementation does not yet support hardware-accelerated kernels like Flash-Attention, making it slower to train in terms of wall-clock time compared to highly optimized standard Transformers.
  • Hyperparameter Complexity: Introducing a second type of attention head adds new hyperparameters (e.g., the ratio of sensory to relational heads) that require tuning.
  • Mechanistic Interpretability: While initial visualizations are promising, more work is needed to understand the specific "circuits" DAT forms to solve complex reasoning tasks.
  • Scaling: The authors suggest exploring DAT in even larger vision-language models where complex scene parsing and object relationships are critical.

Summary updated

Abstract

Relational reasoning is a central component of generally intelligent systems, enabling robust and data-efficient inductive generalization. Recent empirical evidence shows that many existing neural architectures, including Transformers, struggle with tasks requiring relational reasoning. In this work, we distinguish between two types of information: sensory information about the properties of individual objects, and relational information about the relationships between objects. While neural attention provides a powerful mechanism for controlling the flow of sensory information between objects, the Transformer lacks an explicit computational mechanism for routing and processing relational information. To address this limitation, we propose an architectural extension of the Transformer framework that we call the Dual Attention Transformer (DAT), featuring two distinct attention mechanisms: sensory attention for directing the flow of sensory information, and a novel relational attention mechanism for directing the flow of relational information. We empirically evaluate DAT on a diverse set of tasks ranging from synthetic relational benchmarks to complex real-world tasks such as language modeling and visual processing. Our results demonstrate that integrating explicit relational computational mechanisms into the Transformer architecture leads to significant performance gains in terms of data efficiency and parameter efficiency.
BibTeX
@inproceedings{altabaa2025disentangling,
  author = {Awni Altabaa and John Lafferty},
  title = {{Disentangling and Integrating Relational and Sensory Information in Transformer Architectures}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=lbrqeIipJr},
}
Figure