Abstract
Pre-trained Language Models (PLMs) often struggle with abstract reasoning, failing to generalize from specific instances to broader concepts. This paper introduces HiCon-EG (Hierarchical Conceptual Entailment Graph), a framework that enhances PLMs' abstract inference by organizing arguments into hierarchical conceptual structures. By learning entailment across multiple levels of abstraction—from specific entities like "pasta" to broader categories like "food"—HiCon-EG enables models to perform more robust common-sense reasoning. Experimental results demonstrate that HiCon-EG significantly outperforms state-of-the-art models, including GPT-4, on abstract reasoning benchmarks.
Introduction
While Pre-trained Language Models (PLMs) are impressive, they often stumble when moving from concrete instances to abstract concepts—a cornerstone of human cognition. For example, a human knows that if "Mrs. Thompson gives her children pasta," it implies the children "received food" and are now "full."
Existing "entailment graphs," which map how one event leads to another, usually focus on verbs but treat the objects (arguments) as flat, static types. They often miss the polysemy and hierarchical nature of nouns. If a model only sees "pasta" as a generic "entity," it misses the richer common-sense inferences available at the "food" level. This paper aims to bridge that gap by teaching models to navigate these levels of granularity.
Methodology
The authors proposed the HiCon-EG framework, which operates in four distinct steps to build a robust conceptual hierarchy:
- Complex-to-Simple (C2S) Open Information Extraction (OIE): Traditional Information Extraction often fails on long, nested sentences. The authors used LLaMa2-7B to decompose complex sentences into simple ones, then "distilled" this capability into a smaller, more stable BERT model to reduce hallucinations and computational costs.
- Hierarchical Concept Building: For every argument (like "toast"), the model builds a hierarchy using WordNet and Wikidata (e.g., toast → bread → food → activity/entity).
- Entropy-based Concept Selection: Picking the right level of abstraction is critical. If too specific, the model cannot generalize; if too broad (calling everything an "entity"), the information becomes useless. The authors used an objective function that balances entropy (frequency/consistency in the data) with a hierarchical-depth regularization term, solved via a Genetic Algorithm.
- Learning the Graph: Finally, they constructed the entailment graph by calculating relevance scores between predicates across different concept levels, following the Distributional Inclusion Hypothesis: if the contexts of event A are a subset of event B, then A likely entails B.
Evaluation and Results
The researchers evaluated their approach across several benchmarks, including ABSPYRAMID-HC (specifically designed for hierarchical conceptual entailment), AbstractATOMIC, and LSOIE.
- Superior Abstraction: HiCon-EG significantly outperformed standard PLMs and even some LLMs on the ABSPYRAMID-HC dataset. A DeBERTa-large model enhanced with HiCon-EG achieved an 81.88% accuracy, outperforming GPT-4 (65.30%) and Llama3-8B (78.23%).
- OIE Breakthrough: Their C2S extraction method set a new State-of-the-Art (SOTA) on the LSOIE-wiki dataset with an F1 score of 53.92, proving particularly effective for the long, complex sentences found in news corpora.
- Human-Level Granularity: In human evaluations, the entropy-based selection method picked "moderate granularity" concepts (the "Goldilocks" zone of abstraction) at a rate comparable to GPT-4 but with significantly lower error rates.
Discussion
The core philosophy of HiCon-EG is that inference acts as a teacher. By learning that Event A entails Event B at a specific conceptual level, the PLM inherently gains a better understanding of the nouns involved.
A key innovation is the "squeeze" on abstraction via the regularization term. By mathematically forcing the selection process away from overly broad terms like "thing" and toward moderate granularity, the model identifies concepts that provide the most inferential power. Furthermore, distilling LLM logic into a BERT-based sequence labeler provided a 95% accuracy rate and much higher stability for downstream graph construction compared to using raw LLM outputs.
Conclusion
HiCon-EG demonstrates that hierarchical conceptualization is a powerful tool for improving the abstract reasoning of language models. While the current method is largely unsupervised and focuses on entailment, it provides a foundation for more sophisticated common-sense reasoning. Future work will explore integrating supervised data to refine relationship accuracy and expanding the framework to handle a more diverse range of semantic relations beyond simple entailment.
Summary updated