Abstract
We introduce RelNet: a new model for relational reasoning. RelNet is a memory augmented neural network which models entities as abstract memory slots and is equipped with an additional relational memory which models relations between all memory pairs. The model thus builds an abstract knowledge graph on the entities and relations present in a document which can then be used to answer questions about the document. It is trained end-to-end: only supervision to the model is in the form of correct answers to the questions. We test the model on the 20 bAbI question-answering tasks with 10k examples per task and find that it solves all the tasks with a mean error of 0.3%, achieving 0% error on 11 of the 20 tasks.
Introduction
Reasoning about how entities—such as people, places, and objects—interact is a cornerstone of general intelligence. Historically, AI systems approached this through "cascaded pipelines" that involved discrete steps: entity extraction, coreference resolution, relationship identification, and finally, inference. However, these pipelines are fragile; errors in early stages, such as a misidentified entity, propagate and amplify, leading to incorrect final answers.
Recent end-to-end models like Recurrent Entity Networks (EntNet) attempted to mitigate this by tracking entity states in memory slots. While effective for simple queries (e.g., "Where is Mary?"), these models often struggle with complex relational logic because they lack a dedicated mechanism to reason about the connections between entities. RelNet addresses this gap by explicitly modeling both entities and their relationships within a unified memory-augmented framework.
Methodology
RelNet processes documents and answers questions through three primary components: an input encoder, a dynamic relational memory, and an output module. The input encoder reads sentences and converts them into fixed-length vectors using a learned multiplicative mask, allowing the model to focus on relevant keywords.
Dynamic Relational Memory
The core innovation of RelNet is its dual-structured memory. Unlike previous models that focus solely on entity "nodes," RelNet maintains two distinct sets of memory slots:
- Entity Memory ($e_j$): Slots that track the state of individual entities.
- Relational Memory ($r_{ij}$): A set of $D^2$ slots representing the relationship between every possible pair of entities.
As the model reads each sentence ($s_t$), a gating mechanism ($g_i, g_{ij}$) performs a "soft selection" to decide which entities and relationships require updates. This process effectively constructs a "latent knowledge graph" where entities are nodes and relational memories are edges.
Training and Inference
The model is trained entirely end-to-end. It is not provided with explicit labels for entities or relations; instead, it learns to populate its memory slots based solely on the supervision provided by correct answers. For inference, an attention-based output module compares the question vector ($q$) against the concatenated entity and relation vectors to produce the final answer.
Experiments and Results
The authors evaluated RelNet on the 20 bAbI tasks, a standard benchmark for reasoning that requires skills such as path finding, inductive reasoning, and time-based logic. RelNet achieved state-of-the-art performance, solving all 20 tasks (defined as achieving less than 5% error on each).
As shown in the results, RelNet achieved a mean error of 0.3%, outperforming EntNet's 0.5%. It achieved perfect scores (0% error) on 11 out of 20 tasks and demonstrated significant improvements in tasks requiring complex link analysis, such as "compound coreference" and "path finding."
Discussion and Future Work
RelNet's success stems from its explicit modeling of edges in a graph-structured representation, providing a strong inductive bias for relational data. Unlike models that scale with the number of sentences, RelNet scales with the number of entity slots, making it more efficient for long-form text with a consistent cast of characters.
Despite these strengths, several challenges remain. The relational memory grows quadratically ($D^2$) with the number of entity slots, which may be prohibitive for documents with hundreds of unique entities. Furthermore, while the model builds a latent graph, the relational vectors are not inherently human-readable. Future research could focus on "decoding" these memories into human concepts and moving beyond synthetic datasets to massive, noisy real-world corpora.
Summary updated