Abstract
Logical reasoning in natural language processing (NLP) is hindered by the uncertainty of textual information and the lack of natural logical operators in standard Euclidean vector spaces. We present Beta-LR, a novel framework that embeds text as Beta distributions. By leveraging the bounded nature of the Beta distribution, we define interpretable neural operators for intersection, negation, and union. Beta-LR achieves state-of-the-art results on ReClor and LogiQA, particularly excelling on complex "HARD" reasoning subsets while maintaining high parameter efficiency.
Introduction
Traditional NLP models typically embed text into low-dimensional vectors in Euclidean space. However, this approach faces two major challenges in logical reasoning:
- Uncertainty: Logical information in text is often ambiguous. A fixed vector (point estimate) struggles to represent the range of possible interpretations or the density of logical truth.
- Incompatible Operations: Logical operations like AND (intersection), OR (union), and NOT (negation) do not have natural equivalents in standard vector arithmetic.
Beta-LR addresses these by shifting the reasoning process from Euclidean space to the space of probability distributions, specifically the Beta distribution.
Methodology
The Beta-LR framework maps each sentence to a set of Beta distributions defined by shape parameters $\alpha$ and $\beta$. Since the Beta distribution is bounded between [0, 1], it is uniquely suited for representing fuzzy logical values.
Neural Logical Operators
The core innovation of Beta-LR lies in its interpretable neural operators:
- Intersection ($\mathcal{I}$): Modeled as a weighted product of Probability Density Functions (PDFs), identifying shared logical information across sentences.
- Negation ($\mathcal{N}$): Defined by taking the reciprocal of the Beta parameters ($1/\alpha, 1/\beta$), effectively flipping high-probability regions to low-probability regions.
- Union: Derived via De Morgan’s Laws ($A \cup B = \neg(\neg A \cap \neg B)$), ensuring the model adheres to symbolic logic principles.
Integration Process
The model follows a three-step integration: extracting shared information, updating individual sentence representations based on that context, and performing a logical union to create a comprehensive context embedding for prediction.
Experimental Results
Beta-LR was evaluated on two major benchmarks: ReClor (standardized test data) and LogiQA (civil service examination logic).
Performance Gains
- ReClor: The model achieved a 7.2% increase in test accuracy over the RoBERTa baseline. Notably, it showed a 12.0% improvement on the "HARD" subset, demonstrating superior performance in complex reasoning scenarios.
- LogiQA: Accuracy increased by 15.3% on the test set.
- Efficiency: Beta-LR adds only ~5 million parameters to the RoBERTa-large backbone (a 1.4% increase), proving more efficient than many graph-based alternatives.
Key Insights
Ablation studies confirmed that Euclidean vectors are insufficient for logic; replacing Beta distributions with standard vectors led to a significant performance drop (e.g., from 59.6% to 56.2% on ReClor). Furthermore, the model naturally quantifies uncertainty: a strong correlation (0.762) was found between sentence length and the entropy of its Beta distribution.
Conclusion
Beta-LR bridges the gap between neural networks and symbolic logic by using probabilistic embeddings. While the current model focuses on global sharing across sentences, future work will explore local sharing between specific sentence pairs and scaling these probabilistic methods to longer documents.
Summary updated