Abstract
Information extraction techniques automatically create structured databases from unstructured data sources, such as the Web or newswire documents. Despite the successes of these systems, accuracy will always be imperfect. For many reasons, it is highly desirable to accurately estimate the confidence the system has in the correctness of each extracted field. The information extraction system we evaluate is based on a linear-chain conditional random field (CRF), a probabilistic model which has performed well on information extraction tasks because of its ability to capture arbitrary, overlapping features of the input in a Markov model. We implement several techniques to estimate the confidence of both extracted fields and entire multi-field records, obtaining an average precision of 98% for retrieving correct fields and 87% for multi-field records.
Introduction
Information Extraction (IE) systems—tools that transform unstructured text into structured database entries—are essential for large-scale data processing. However, these systems are inherently imperfect. Errors in extraction can propagate through databases, leading to compromised data integrity and flawed downstream analysis. While most IE models are designed to identify the most likely sequence of labels, they often lack a principled mechanism to quantify the reliability of their predictions.
Furthermore, while previous research has often focused on token-level confidence, real-world applications require confidence scores for higher-level semantic units: fields (e.g., a multi-word "Job Title") and records (e.g., a complete contact card). This paper presents a mathematically sound approach to confidence estimation for these multi-word units using Conditional Random Fields (CRFs).
Methods
The proposed methodology leverages Conditional Random Fields (CRFs), a discriminative probabilistic framework for sequence labeling. The core innovation is the application of the Constrained Forward-Backward (CFB) algorithm to calculate confidence.
In a standard CRF, the Forward-Backward algorithm computes the total probability of all possible label sequences. To determine the confidence of a specific extracted field, the authors "constrain" the algorithm by locking in the labels for the tokens within that field. The confidence score is then the ratio of the probability of all paths consistent with the extraction to the total probability of all possible paths. This provides a normalized value between 0 and 1.
Additionally, the authors explored Maximum Entropy (MaxEnt) Reranking. This involves training a secondary classifier to predict extraction correctness based on features such as field length, token-level probabilities, and the CFB score itself.
Experimental Setup
The methods were evaluated on a dataset of 2,187 contact records (over 27,000 words) sourced from emails and the web. The data includes 25 semantic classes, such as FirstName, CompanyName, and AddressLine. The CRF was trained on 60% of the data, with the remainder used for development and testing.
Results and Analysis
The CFB method demonstrated high efficacy in distinguishing correct extractions from errors. For individual fields, the system achieved an average precision of 98%. For entire multi-field records, which present a significantly harder challenge, the system achieved an average precision of 87%.
As shown in the accuracy-coverage plot, the CFB and MaxEnt methods closely track the "Optimal" theoretical performance, significantly outperforming simpler heuristics like "Gamma" (which sums individual word probabilities) or random selection.
The record-level results highlight that CFB and MaxEnt provide the strongest correlation (Pearson’s $r$) with actual correctness. Interestingly, the MaxEnt reranker did not significantly outperform the base CFB method. This suggests that because CRFs are discriminative models already trained to maximize conditional probability, there is limited room for a secondary classifier to capture additional signal.
Discussion and Future Work
While the CFB algorithm is mathematically robust and highly accurate, it introduces additional computational overhead compared to simple Viterbi decoding. This trade-off is generally acceptable given the critical need for reliable confidence scores in automated systems.
These scores are particularly valuable for "human-in-the-loop" or interactive IE systems, where human effort can be prioritized on the least-confident extractions. Future research could investigate the impact of these confidence measures on user productivity and explore their application in more complex, non-linear graphical models beyond linear-chain CRFs.
Summary updated