Compute the Interactions, Not the Attributes: What a Fact Layer Must Supply in the Chinese Sexagenary System

Huixue Han, Jiaying Liu

TL;DR

When pairing LLMs with external "fact layers" (like RAG or tools), providing raw attributes isn't enough—models need pre-computed interactions. This paper demonstrates that while LLMs struggle to combine static facts into complex relations, providing an exhaustive list of all possible relations often confuses them; the optimal strategy is a query-filtered layer that supplies only the specific interactions requested.

What Problem Does This Paper Address?

In rule-governed domains—ranging from legal analysis to complex games—we often supplement LLMs with a "fact layer" or tool that provides deterministic data. However, there is little consensus on what that layer should actually return. Should it provide raw "attributes" (e.g., "this piece is a Rook"), every possible "relation" (e.g., every attack on the board), or only the specific "answer" to the query?

The authors tackle this using the Chinese sexagenary system (ganzhi) and BaZi chart analysis. This domain is ideal because it is strictly deterministic and table-checkable, yet involves multi-step reasoning where errors in one step (like identifying a "day master") cascade into a completely wrong interpretation. The goal is to determine the "ladder" of factual support required to move models from near-zero accuracy to reliable performance.

Methods

The researchers evaluated 11 open-weight models (including the Qwen3 series, Gemma, and OLMo) and 3 proprietary reasoning models (Claude 3.5 series). They tested these models across a "fact-layer ladder" consisting of four rungs:

  1. Closed: The model answers from its internal weights only.
  2. +attrs: The model is given static attributes (e.g., the element and polarity of a pillar).
  3. +full: The model is given a complete list of all possible relations in the chart (high noise/distractors).
  4. +rel: A query-filtered layer that provides only the relations the specific question asks about.

They also compared "no-thinking" (greedy decoding) vs. "thinking" (chain-of-thought) modes to see if internal reasoning could substitute for external computation. To ensure the findings weren't specific to Chinese culture or language, they replicated the experiment using a Chess-based dataset in English.

Figure 1: The fact-layer ladder on one collision item (transit pillar given). Each rung supplies more of the com- putation to the same model; bars give exact accuracy over 80 items (orange: Qwen3-32B without thinking; blue: Qwen3.8-27B with thinking). The complete output (+FULL) contains every relation the answer needs, mixed with relations between natal pillars that the question does not concern; stating only the relations it concerns (+REL.) takes Qwen3-32B from 1% to 92%.

Datasets

The authors introduced a new benchmark consisting of:

  • 682 items across 12 fact categories in the Chinese sexagenary system (from simple look-ups to 4-step derivation chains).
  • 160 transit–natal collision items, which require computing how a specific day interacts with a birth chart.
  • 160 Chess items, mapping pieces and their spatial relations (attacks, defenses, etc.) to the same structural logic as the BaZi collisions.

All gold answers were cross-checked against independent software implementations to ensure 100% factual accuracy.

Key Insights and Ideas

  • Attributes are not "Building Blocks": Simply providing the raw ingredients (attributes) does not help models compute the final result. In both BaZi and Chess, adding attributes had a negligible or even negative effect on accuracy.
  • The "Full List" is a Distractor Trap: Providing every possible relation (the +full rung) often leads to failure in direct-answering models. Models frequently "hallucinate" relations by misattributing a relation between two background entities to the entity mentioned in the query.
  • Filtering is the Secret Sauce: For models answering directly, a query-filtered layer is transformative. For example, Qwen3-32B jumped from 1% to 92% accuracy just by removing irrelevant relations from the context.
  • Reasoning Models are Different: Proprietary "thinking" models (like Claude 3.5) are much better at using the +full list or even the raw reference tables, but they still benefit from the filtered layer to reach a 100% ceiling.
Figure 2: Closed-book accuracy (%) by category (ab- breviations as in Table 1), grouped by the steps an an- swer needs. Top row: majority-class answer; apricot and bold: at or below it. Look-ups rise with model gen- eration and size; the day pillar and the transit ten god stay at or below the majority answer for every model except the two Qwen3.8 models with thinking.

Key Results

  • Direct Answering Failure: In the "Closed" condition, no open-weight model got more than 4% of sexagenary days or 1% of pillar collisions right.
  • The Power of +REL: For Qwen3-8B and 32B, the query-filtered layer raised accuracy to 78–92%. Four other models reached 96–100% with this support.
  • Chess vs. BaZi: While the "complete list" was more usable in Chess than in BaZi (likely due to Chess being more prevalent in training data), the filtered layer still outperformed the complete list in the majority of test cases. (See Figure 4).
Figure 4: Without thinking, the complete list (hollow) already lifts 9 of 11 models to at least 21% in chess, while 7 of 11 stay below 2% in BaZi. (a) Collision items with the transit pillar given; (b) chess items. *: +REL. beats +FULL after Holm correction (C2).

Limitations and Future Work

  • Perfect Filtering Assumption: The +rel condition assumes a "perfect filter" already knows what the user is asking. In practice, building a tool that accurately identifies the relevant subset of facts is its own challenge.
  • Deterministic Focus: The study focuses on rule-governed, "hard" logic. It remains to be seen how these findings translate to "soft" domains like sentiment analysis or creative writing.
  • Layout Sensitivity: The authors note that the gain from the filtered layer comes from a combination of reduced length, better layout, and the use of explicit "none" markers. Future work could disentangle which of these formatting choices matters most.
  • Reasoning Overrides: Interestingly, some "thinking" models occasionally re-derived facts and overrode the correct information provided in the prompt. Systems pairing reasoning models with fact layers should include checks to ensure the model doesn't "reason its way" into an error.

Summary updated

Abstract

Language-model systems in rule-governed domains often pair the model with a deterministic fact layer, but which facts that layer should supply is rarely measured. We measure it in the Chinese sexagenary system underlying BaZi chart analysis, where every fact has one table-checkable value: 682 items over twelve fact categories and 160 transit--natal collision items, posed closed-book, with the reference table, and with computed facts at increasing levels. In eleven open-weight models answering directly, errors concentrate where attributes must be combined: none gets more than 4\% of sexagenary days or 1\% of collisions right. Static attributes have no consistent effect, and the complete relation list leaves most direct-answering models near zero although it contains the answer. A query-filtered layer (exploratory) raises Qwen3-8B and Qwen3-32B to 78--92\% and four further models to 96--100\%. Three proprietary reasoning models compute 68--95\% of collisions unaided. On chess positions, under predictions locked in advance, attributes again do not substitute for relations and listed distractors are misattributed. The fact layer must supply computed interactions; how they are presented depends on the model and the domain.
BibTeX
@misc{han2026compute,
  author = {Huixue Han and Jiaying Liu},
  title = {{Compute the Interactions, Not the Attributes: What a Fact Layer Must Supply in the Chinese Sexagenary System}},
  year = {2026},
  eprint = {2611.13597},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13597},
}
Figure