SymbolicThought: Integrating Language Models and Symbolic Reasoning for Consistent and Interpretable Human Relationship Understanding

Runcong Zhao, Qinglin Zhu, Hainiu Xu, Bin Liang, Lin Gui, Yulan He

Introduction

Mapping interpersonal relationships within complex narratives—ranging from historical archives to legal documents and literature—is a cornerstone of research in the humanities and social sciences. Traditionally, this process relies on manual annotation, a labor-intensive task that is often slow and prone to overlooking long-distance or implicit connections. While Large Language Models (LLMs) present a potential solution for automating relationship extraction, they frequently suffer from "hallucinations" and logical inconsistencies. For instance, an LLM might contradict itself by labeling a character as both a father and a son to the same person, or by failing to recognize the reciprocal nature of a friendship.

To address these challenges, we present SymbolicThought, a hybrid framework that integrates the broad extraction capabilities of LLMs with the rigorous logic of symbolic reasoning. By employing an "Explorer-Verifier" pattern and a human-in-the-loop interface, SymbolicThought produces character relationship graphs that are more accurate, consistent, and interpretable than those generated by standalone AI or manual human effort.

The SymbolicThought Framework

SymbolicThought operates through a multi-stage pipeline designed to maximize both recall and logical coherence. The process begins with character extraction and proceeds through automated reasoning and interactive refinement.

The SymbolicThought multi-stage pipeline and user interface.

1. Character Extraction and Verification

In the first stage, an LLM "Explorer" identifies potential characters. By using a high "temperature" setting, the model performs multiple passes to ensure high recall. A frequency filter then isolates the most consistent names. Finally, a human user verifies the list, resolving coreferences and aliases (e.g., "Queen Elizabeth" vs. "Her Majesty") to ensure a clean entity set.

2. Relation Extraction and Symbolic Reasoning

Once characters are established, the system predicts pairwise relationships. The defining feature of SymbolicThought is its Symbolic Reasoning Module, which evaluates these predictions against seven distinct types of logical constraints. These constraints are categorized into inferable and conflicting relationships.

Logical constraints used in the Symbolic Reasoning Module to ensure consistency.
  • Inferable Relationships: These include Symmetry (if A is a friend of B, B is a friend of A), Inversion (if A is B's husband, B is A's wife), Composition (transitive relations like grandparenthood), and Hierarchy (if A is an elder brother, A is also a brother).
  • Conflicting Relationships: These identify logical impossibilities, such as Incompatible (A cannot be both husband and wife to B), Asymmetric (A cannot be B's husband if B is A's daughter), and Exclusive (limiting the number of specific roles, such as a single husband).

3. Conflict Resolution via RAG

When the symbolic module detects a logical clash, the system employs Retrieval-Augmented Generation (RAG). It pulls relevant snippets from the source text and presents the LLM with a multiple-choice prompt based on that evidence. This "self-correction" mechanism allows the model to break internal biases by grounding its decisions in specific textual evidence.

Experimental Setup and Datasets

The framework was evaluated using several benchmarks, including a novel Relationship Logic Dataset of 160 interpersonal relationships and a Narrative Corpus of 19 diverse texts spanning historical accounts, news reports, biographies, and fiction. These were compared against a gold-standard set of 1,398 expert-annotated relation triplets.

Key Insights and Results

Our analysis revealed a significant "Logic Gap" in current LLMs. Even advanced models like GPT-4 struggle with basic relational logic, particularly in identifying conflicting relations. However, SymbolicThought bridges this gap effectively.

Performance and Efficiency

SymbolicThought consistently outperformed baselines such as Direct Prompting and Self-Consistency. On the "Add" task (inferring new relations), it achieved an F1-score of 37.9, surpassing GPT-4’s 33.4. Furthermore, the symbolic layer ensured that the final graphs were 100% logically consistent—a result no standalone LLM could replicate.

The framework also demonstrated remarkable human-AI synergy. It achieved 25% higher recall than manual human annotation alone and reduced average annotation time by up to 40%. For biographies, the time required dropped from 87 minutes to just 45 minutes.

The Small-World Advantage

A notable finding was the correlation between a text's "Small-World Index" (SWI) and annotation ease. Denser social networks provide more "logical anchors," allowing the symbolic module to more easily catch errors and infer missing links through composition and symmetry.

Conclusion

SymbolicThought demonstrates that the future of complex narrative analysis lies in the integration of LLMs with structured symbolic logic. While challenges remain—such as capturing implicit social common sense and resolving directionality errors—the framework provides a robust foundation for large-scale, consistent relationship mapping. Future work will explore temporal reasoning to track how relationships evolve over time and apply the system to massive archives like legal or literary collections.

Summary updated

BibTeX
@inproceedings{zhao2026symbolicthought,
  author = {Runcong Zhao and Qinglin Zhu and Hainiu Xu and Bin Liang and Lin Gui and Yulan He},
  title = {{SymbolicThought: Integrating Language Models and Symbolic Reasoning for Consistent and Interpretable Human Relationship Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.4},
}
Figure