TL;DR
This paper tackles the challenge of teaching neural language models to understand the quantitative aspects of numbers in text, not just their symbolic presence. They introduce a novel loss function and a regression-based approach to improve how models predict numerals, showing that considering numerical magnitude leads to more accurate predictions of quantities.
What Problem Does This Paper Address?
Current neural language models, like BERT and GPT, treat numbers as mere string tokens, similar to any other word. This approach fails to capture the inherent quantitative nature of numerals – their size, weight, or value. For instance, understanding that "200 cm" implies a tall person requires more than just recognizing the digits; it demands numerical common sense (NCS). This lack of quantitative understanding limits models' ability to perform tasks requiring numerical reasoning, error detection, or even basic comprehension of texts involving quantities. The paper aims to bridge this gap by developing methods to imbue language models with a better grasp of numerical magnitudes.
Methods
The authors propose two main approaches to enhance numerical common sense in language models:
- LossNUM: This is a novel loss function designed for fine-tuning masked word prediction (MWP) models. Instead of just penalizing incorrect predictions, LossNUM assigns a larger penalty for predictions that are numerically further from the groundtruth numeral. It achieves this by incorporating the logarithmic difference between predicted and groundtruth numerals into the standard cross-entropy loss. This encourages models to predict numerals that are closer in magnitude, even if not an exact match.
- REG Model: This approach reframes the masked numeral prediction task as a regression problem. A BERT model is augmented with an additional output layer that directly predicts a numeral's magnitude. This model is trained using a Mean Squared Error (MSE) loss, also on the logarithmic scale (LossMSE), to prioritize accurate magnitude prediction.
Both approaches treat numerals as single tokens, avoiding the potential loss of meaning when splitting them into sub-words.
Datasets
The experiments were conducted on four diverse datasets:
- DROP (Wikipedia): A dataset for reading comprehension requiring numerical operations.
- arXiv (Science Papers): Contains scientific articles.
- FinNum (Financial Tweets): Focuses on financial data.
- Numeracy-600K (Article Titles): Includes market comments.
These datasets vary in passage length, domain, and the distribution of numerals, allowing for a comprehensive evaluation.
Key Insights and Ideas
- Two Facets of Numerical Understanding: The paper highlights that numerals have both a symbolic aspect (e.g., "7" is a specific token) and a quantitative aspect (e.g., "7" represents a magnitude). Existing models excel at the former but struggle with the latter.
- Magnitude Matters for Prediction: Simply predicting the correct numeral token isn't enough; predicting a numeral that is numerically close to the groundtruth is often more valuable, especially for quantity-type NCS.
- Loss Function Design is Crucial: A standard cross-entropy loss treats all incorrect predictions equally. The proposed LossNUM, by penalizing larger numerical errors more heavily, effectively guides models towards better quantitative understanding.
- Different Models for Different Numerals: The REG model, while having lower overall accuracy, shows promise in predicting larger numerals, suggesting that hybrid approaches might be beneficial.
Key Results
- LossNUM Reduces Absolute Errors: Fine-tuning with LossNUM significantly reduced numerical absolute errors (MdAE and MdAPE) on datasets with quantity-type numerals (SP and FT), demonstrating its effectiveness in capturing quantitative aspects.
- Exact Match Accuracy Trade-off: While LossNUM improved quantitative accuracy, it sometimes led to a slight decrease in exact match accuracy, particularly on datasets with more token-type numerals (WP and AT).
- REG Model's Strengths and Weaknesses: The REG model showed lower overall prediction accuracy but was better at predicting large numerals and exhibited fewer large errors compared to standard MWP models.
- Contextual Clues are Important: The paper notes that models can sometimes infer numerical ranges from context, but this understanding is limited without explicit quantitative training.
Limitations and Future Work
- Evaluation Metrics: The authors acknowledge that MdAE and MdAPE, while useful, can be sensitive to the scale of numerals and may not perfectly capture all aspects of numerical prediction. They suggest further consideration for appropriate evaluation metrics.
- Hybrid Models: The REG model's complementary strengths to MWP models suggest that combining these approaches could lead to improved overall performance. This is identified as a direction for future work.
- Token-Type vs. Quantity-Type: The performance differences across datasets highlight that models' ability to handle token-type numerals (like dates) versus quantity-type numerals (like measurements) varies, indicating a need for more nuanced approaches.
Summary updated