Learning Faithful Representations of Causal Graphs

Ananth Balashankar, Lakshminarayanan SubramanianACL 2021 · OpenReview

TL;DR

This paper introduces a new definition of "faithfulness" for text embeddings, ensuring they accurately reflect the directed relationships in causal graphs. By incorporating this faithfulness property into BERT and RoBERTa models, the authors achieve significant improvements in causal question-answering tasks, demonstrating that embeddings can indeed learn to represent causal structures.

What Problem Does This Paper Address?

Current methods for learning causal representations from text often focus on predicting direct causal links. While useful, this approach can miss indirect causal relationships, leading to spurious correlations in downstream tasks like causal effect estimation. The challenge is to create text embeddings that not only capture direct cause-effect pairs but also preserve the overall directed structure and geometric properties of a causal graph, enabling more robust causal reasoning.

Methods

The core of this work is the definition of a "faithfulness" property for embeddings. This property ensures that the geometric relationships (like distance) in the embedding space mirror those in the causal graph. Specifically, they propose three conditions:

  • Neighborhood Preservation: Similar to how nodes in a causal graph have neighbors, the embeddings should group related nodes together.
  • Uniformity: The embedding space should be uniformly distributed, preventing biases.
  • Distance Correlation: Inequalities in distances between nodes in the causal graph should be preserved in the embedding space.

To achieve this, they fine-tune pre-trained models like BERT and RoBERTa. They introduce a novel quasi-pseudo distance metric that captures the directed nature of causality (i.e., the distance from cause to effect is different from effect to cause). This metric, along with regularization terms for neighborhood, uniformity, and distance correlation, is incorporated into the training objective. The model is trained to minimize violations of these faithfulness properties while also performing causal link prediction.

Datasets

  • Gigaword Causal Graph: A dataset derived from the Gigaword corpus, containing approximately 800K causal links.
  • ClueWeb12 CauseNet: A larger causal graph extracted from the ClueWeb12 web crawl, with around 200K causal links, validated by human annotators.
  • Yahoo! Answers Corpus: A collection of 3031 causal questions of the form "What causes X?", used for evaluating the downstream question-answering task.

Key Insights and Ideas

  • Faithfulness is More Than Link Prediction: Simply predicting direct causal links isn't enough. Embeddings need to capture the geometry of the causal graph to be truly useful for causal reasoning.
  • Directed Distance Matters: Standard symmetric distance metrics (like cosine similarity) fail to capture the directionality inherent in causal relationships. A specialized quasi-pseudo metric is crucial.
  • Regularization for Structure: Incorporating regularization terms that enforce neighborhood, uniformity, and distance correlation properties directly into the embedding learning process is key to achieving faithfulness.
  • Transferable Knowledge: Faithfulness learned during embedding training directly translates to improved performance on downstream causal inference tasks, like answering "what causes X?" questions.

Key Results

  • Improved Faithfulness Metrics: The proposed "Faithful-BERT" and "Faithful-RoBERTa" models showed significant improvements (over 30%) in neighborhood, uniformity, and distance correlation metrics compared to baselines.
  • Enhanced Neighborhood Detection: The area under the precision-recall curve for detecting neighboring nodes in the causal graph increased by 21-23% for Gigaword and 17-20% for CauseNet.
  • Causal QA Performance Boost: In the Yahoo! Answers causal QA task, Faithful-RoBERTa achieved a precision of 41.07% (P@1), outperforming existing baselines by 10.2%. The mean reciprocal rank also improved by 10.8%.
  • Qualitative Re-alignment: Analysis showed that causal word pairs moved closer in the faithful embedding space, while associative but non-causal pairs moved further apart, indicating a successful re-alignment towards causal structures.

Limitations and Future Work

  • Assumption of Causal Graph Availability: The work assumes a pre-existing, human-validated causal graph. Building such graphs is a significant challenge in itself.
  • Computational Cost: Fine-tuning large transformer models with these additional regularization constraints can be computationally intensive.
  • Future Work: The authors suggest exploring how to learn these faithful embeddings without relying on a pre-existing causal graph, and extending the approach to handle more complex causal structures beyond directed acyclic graphs.

Summary updated

BibTeX
@inproceedings{balashankar2021learning,
  author = {Ananth Balashankar and Lakshminarayanan Subramanian},
  title = {{Learning Faithful Representations of Causal Graphs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=VA2eR7HhtP},
}
Figure