Large language models are increasingly used as assistants for causal discovery, the task of inferring which variable causes which from observed data. A typical setup has one model read the variable names and background text while a statistics routine looks at the numbers. The model then edits the graph the statistics produced. This setup shares a failure with vision-language models, which are known to ignore the image and answer from language habits. In both cases, whenever the text and the numbers disagree, the model will likely side with the text. We propose to measure this "text dominance" directly, using data whose true causal direction is set by us and deliberately made to contradict what the names suggest. We then test a fix built from two parts. The first is a task decomposition in which one agent sees only the text and another sees only the anonymized data. The second is a non-linear causal test that gives the data agent a real signal about direction.
The opportunity exists because the best tool-augmented agent system for causal discovery, MATMCD, feeds text-derived constraints back into linear or constraint-based statistical methods such as PC and DirectLiNGAM. It never asks whether its language models override the data. Its statistical engine also cannot orient an edge between two variables whose relationship is non-linear with arbitrary noise. That is exactly the regime where non-linear methods can find the direction from observational data alone. If agents mostly follow names, their benchmark gains may come from domain knowledge the model memorized, and they will fail on the cases where discovery matters most: new or surprising mechanisms.
Core Claim
We make two claims. The first is diagnostic: when a single language model receives both variable descriptions and data evidence, its decisions about edge direction follow the text channel far more often than the strength of the data evidence justifies. This is the causal-discovery counterpart of the text-prior dominance documented in vision-language models. The second is constructive: this bias can be removed without losing the benefit of domain knowledge. The recipe is to blind each channel from the other and then combine them with a fixed, calibrated rule. The combination should not be done by letting a model read both channels, and it should not be done by agent debate.
The data channel needs its own real signal of direction. Non-linearity supplies one. Under an additive noise model, meaning the effect is a non-linear function of the cause plus noise independent of the cause, written $Y = f(X) + \epsilon$ with $\epsilon$ independent of $X$, the leftover error is independent of the input only when the regression runs in the true causal direction. Least-squares independence regression (LSIR) fits the regression by directly minimizing an estimate of the dependence between input and residual. It chooses its settings by cross-validation, and it identified 49 of 80 real cause–effect pairs, against 31 for the kernel-based alternative. LSIR is the method we reuse for the data agent.
Research Program
The first component is a conflict benchmark. We start from the 80 real-world cause–effect pairs used by LSIR, keeping each pair's real variable names and descriptions. For every pair we build two semi-synthetic twins. We fit additive noise models on the real data to obtain realistic functions and noise shapes. From those, one twin's data is generated so its true direction agrees with the named direction, and the other's so its true direction contradicts it. In the conflict twin, "altitude" is made to depend on "temperature."
The same construction applies to the multi-variable graphs in the MATMCD suite (AutoMPG, Sachs protein, Asia, Child). There we reverse a random subset of edges in a ground-truth graph and generate non-linear data that matches the reversed graph.
On this benchmark we define a text-reliance index: the fraction of conflict items on which a system outputs the named direction, reported separately for each level of data-evidence strength. We measure evidence strength by LSIR's gap in residual dependence between the two directions. An unbiased system should follow the data more often as evidence grows stronger. A text-dominated system will show a flat curve.
The second component probes the failure where it is most familiar. We show the same conflict pairs to a vision-language model as scatter plots, with and without axis labels. This connects the finding to the known imbalance in which models give visual tokens only about a quarter of the attention in deep layers (LLaVA Steering; Re-Balancing Contrastive Decoding). The labeled-versus-unlabeled gap on conflict items tells us whether plotted data is overridden by labels the way pixels are overridden by language priors.
The third component is the decomposed system. It has three pieces:
- Text agent. Sees only names and retrieved descriptions. It outputs, for each candidate edge, a probability that the edge points one way, and it may abstain.
- Data agent. Sees only anonymized columns. It runs LSIR in both directions and returns the dependence gap. We turn that gap into a calibrated probability using permutation tests on held-out synthetic pairs. The data agent also runs a rank-based check for hidden common causes in the style of Latent Hierarchical Causal Structure Discovery with Rank Constraints. When that check flags a likely confounder, the data agent marks its own evidence as unreliable, since confounding breaks additive-noise independence.
- Arbiter. A fixed rule, not a language model. It adds the two channels' log-odds, each weighted by that channel's reliability as measured on a small calibration split.
The decomposition does the real work: it enforces blinding, and each agent's tool choice stays inside its own lane. It differs from MATMCD in three ways: its agents see both channels, it relies on linear or constraint-based statistics, and it lets a language model produce the final constraints. It differs from the post-hoc debate system for vision-language hallucination (Post-hoc MAS), which settles disagreements between tool evidence and text by open-ended debate among models. Our proposal replaces that step with a statistical test and a numeric rule. It shares only the multi-agent label with LEGO, whose cause and effect analysts reason purely in natural language and never touch data.
Baselines
The naive reference points are the two single-channel systems. The first is LSIR run alone on anonymized data. On the original 80 pairs it has a reported score of 49/80 correct, with a correct-identification rate of 0.77 at a strict decision threshold (LSIR). It should be unaffected by name conflicts and weak where the additive noise assumption fails. The second is a text-only model given names alone, which should be near-perfect on agreeing twins and near-zero on conflict twins. Any fused system has to beat the better of these two on the combined agreeing-plus-conflict set. Otherwise the fusion adds nothing.
The strongest existing method is MATMCD, run with its published agent prompts on the same language model backbone and the same retrieval budget as ours. It reports up to 66.7% reduction in normalized Hamming distance (the count of wrong, missing, or extra edges, normalized by graph size) and precision 1.00 on AutoMPG. The file gives no absolute graph-error number for its statistical baselines, so we will rerun both ourselves rather than quote. To isolate the statistical engine from the agent design, we also run MATMCD with LSIR substituted for DirectLiNGAM/PC, so both systems get the same non-linear signal.
The ablation that isolates our key ingredient is a single-agent unblinded version. One model receives the names, the LSIR evidence written as text, and the scatter plot, and it decides directly. It gets the same inputs as the full system, just not separated. We also test a debate arbiter, in which two model agents argue from the two channels. That variant checks whether the fixed numeric rule matters.
All systems use identical data, identical language model, and a matched token budget. Every tuned threshold, including ours, is chosen on the same calibration split. Beating the baselines means two things at once: accuracy on conflict twins significantly above the unblinded single agent and adapted MATMCD, and accuracy on agreeing twins and original data no worse than MATMCD's.
Why not just…?
Why not just plug a non-linear test into MATMCD and stop? This is the most serious threat. If the reliance index shows that adapted MATMCD already follows strong data evidence on conflict items, then blinding is unnecessary and our diagnostic becomes a negative result about the imbalance hypothesis. We run exactly this variant and report it either way. The vision-language evidence predicts it will still defer to names. There, models over-weight their own earlier text even when the image contradicts it (Hallucination Snowballing), and "lazy" attention heads favor text tokens roughly fivefold (Modular Attribution and Intervention). That evidence is indirect, though, so the experiment has to settle it.
Why not just combine an LLM prior with the LSIR score by a Bayesian formula, with no agents at all? Honestly, our arbiter is close to this, and we expect most of the gain to come from blinding plus calibrated combination rather than from agent conversation. The agents' remaining job is narrower. The text agent decides when domain knowledge exists at all, abstaining on obscure variables. The data agent decides when its own test is untrustworthy, using the confounding check. We compare against a version with a fixed prior weight and no abstention or reliability flags. If that version matches ours, we will report the contribution as the diagnostic plus the blinding principle, and drop claims about the decomposition.
Why not just use the data alone, since text is the source of bias? On real benchmarks, names usually point the right way. Discarding them throws away accuracy wherever the non-linear test is weak: near-linear or confounded pairs, and the E. coli regulatory links, where LSIR recovered only 3/9 in the open setting (LSIR). The combined agreeing-plus-conflict evaluation shows directly whether fusion beats data-only.
Why not just use contrastive decoding to debias the single model? Contrasting outputs with and without the text is the causal-discovery analogue of Re-Balancing Contrastive Decoding, which lowered sentence-level hallucination on LLaVA-v1.5-7B from 20.8% to 17.8%. That is a modest reduction for a single model that still sees both channels. We include a name-masked contrastive variant of the single agent. If it closes the conflict gap at lower cost, it is the better practical recipe.
Proposed Experiments
The first decisive check is the text-reliance curve on conflict twins. We plot, against data-evidence strength, the probability of following the names for the text-only model, the single unblinded agent, adapted MATMCD, and our system. The imbalance hypothesis predicts flat curves for the unblinded systems and a rising curve for ours. The scatter-plot variant, with and without axis labels, gives the same curve for a vision-language model.
The second check is overall accuracy. We report direction accuracy on the 80 original pairs and their twins, and normalized Hamming distance on the perturbed and original MATMCD graphs.
The third is a stress test in which we inject hidden confounders into the synthetic generator. It asks whether the data agent's reliability flag prevents confident wrong orientations when additive-noise independence no longer holds.
We will count the proposal as successful only if two conditions both hold: conflict-twin accuracy rises clearly above both unblinded baselines, and accuracy on agreeing and original data does not fall below MATMCD.
References
- Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery
- Least-squares independence regression for non-linear causal inference under non-Gaussian noise
- LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
- Mitigating Hallucination in Visual-Language Models via Re-balancing Contrastive Decoding
- Latent Hierarchical Causal Structure Discovery with Rank Constraints
- Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System
- LEGO: A Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for Causality Explanation Generation
- Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
- Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention