Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning

Moxin Li, Fuli Feng, Hanwang Zhang, Xiangnan He, Fengbin Zhu, Tat-Seng Chua

TL;DR

Current Neural Discrete Reasoning (NDR) models excel at answering questions about provided data but fail significantly when asked "what if" questions. This paper introduces Learning to Imagine (L2I), a module that allows models to "imagine" a counterfactual context by identifying specific facts to intervene upon and deriving new values through discrete operators.

The Challenge of Hypothetical Reasoning

Neural Discrete Reasoning (NDR) combines deep learning’s semantic understanding with symbolic logic to answer complex questions over text and tables. While state-of-the-art models perform well on factual questions, they exhibit a "generalization failure" on hypothetical question answering (HQA).

Comparison between factual and hypothetical reasoning contexts.

For example, if a financial report says revenue was $100M, a standard model can tell you the growth rate. But if you ask, *"What would the growth rate be **if** revenue had been $120M?"*, the model's performance collapses. Research shows that a top-tier model's F1 score plummeted from 68.6% to a dismal 3.8% when questions were modified to be hypothetical. The core issue is that these models cannot "imagine" how a change in one variable propagates through the context.

The Learning to Imagine (L2I) Framework

The authors propose the Learning to Imagine (L2I) module, which treats hypothetical reasoning as a two-step causal intervention:

  1. Identifying the Target Fact: Using a tagging head (based on RoBERTa and a matching block with cross-attention), the model scans the factual context to find the specific number or data point that the "if" statement intends to change.
  2. Deriving the Intervention Result: Instead of trying to "hallucinate" a new paragraph, the model uses a set of discrete Deriving Operators (such as SWAP, ADD, MINUS, PERCENT_INC). For instance, if the assumption is "if revenue doubled," the model applies a multiplication operator to the identified fact to create a "counterfactual context."
The L2I architecture integrating RoBERTa, matching blocks, and deriving operators.

This L2I module is designed to be "plug-and-play," sitting between the initial data encoding and the final reasoning stage of existing NDR models like TAGOP.

Datasets and Benchmarking

  • TAT-QA: A pre-existing benchmark for hybrid (tabular and textual) question answering in the financial domain.
  • TAT-HQA (New): The authors introduced this dataset specifically for this paper. They recruited finance students to create 8,283 hypothetical questions based on TAT-QA. Each entry includes a factual context, a hypothetical assumption, and the calculated counterfactual answer. This is a significant new benchmark for the community.

Key Insights

  • Explicit vs. Implicit Imagination: Generative models like GPT-3 struggle with the rigid structure of tables and exact numerical values. Explicitly modeling the intervention via operators is much more robust and precise than unconstrained generation.
  • The "Local Surgery" Principle: Borrowing from Judea Pearl’s causality theory, the model performs "local surgery" on the data. It changes only the specific variable mentioned in the assumption while keeping the rest of the context stable.
  • Semantic Matching: The "Matching Block" in the architecture uses cross-attention to align the words in the hypothetical assumption with the corresponding cells in a table or sentences in a paragraph, ensuring the model intervenes on the correct fact.

Results

The authors integrated L2I into TAGOP, the state-of-the-art model for TAT-QA.

  • Massive Recovery: On the new TAT-HQA dataset, the vanilla TAGOP model achieved only a 41.4% F1 score. With the L2I module, performance jumped to 54.7% F1 (a ~20% relative improvement).
  • Operator Efficiency: The model was particularly effective at the SWAP operator (replacing a value), achieving 60.5% EM (Exact Match).
  • Generalization: By separating "imagination" (L2I) from "calculation" (the NDR head), the model becomes much better at handling unseen hypothetical scenarios.

Limitations and Future Work

Currently, L2I mostly handles "one-step" changes. In the real world, if one number changes, it might have a cascading effect on other totals in a table. The authors suggest "Multi-iteration derivation" as a future path to handle these dependencies. Additionally, expanding the operator library and handling multi-fact interventions (e.g., "If both revenue and costs had doubled...") remain key challenges for future iterations.

Summary updated

BibTeX
@inproceedings{li2022learning,
  author = {Moxin Li and Fuli Feng and Hanwang Zhang and Xiangnan He and Fengbin Zhu and Tat-Seng Chua},
  title = {{Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.5},
}
Figure