ANTS: A Framework for Retrieval of Text Segments in Unstructured Documents

Brian Chivers, Mason P. Jiang, Wonhee Lee, Amy Ng, Natalya I. Rapstine, Alex StorerNAACL 2022

TL;DR

This paper introduces ANTS, a novel framework that leverages pre-trained BERT models to efficiently identify and extract specific text segments from unstructured documents with minimal manual labeling. By using a windowing technique for training and a combined 1-block/5-block score approach for prediction, ANTS significantly reduces the cost and time associated with manual data annotation for researchers.

What Problem Does This Paper Address?

Business and social science researchers often need to extract specific information from large volumes of unstructured text documents. Traditional methods like keyword dictionaries or manual annotation are either too simplistic to capture nuanced language or prohibitively expensive and time-consuming. While advanced NLP models like BERT have shown promise, adapting them to specific extraction tasks typically requires substantial labeled data, which is often unavailable in specialized domains. This paper addresses the challenge of efficiently and accurately identifying relevant text segments in unstructured documents when labeled data is scarce, aiming to democratize the use of advanced NLP for researchers.

Methods

The ANTS framework consists of four main steps:

  1. Labeling: A small set of documents is manually annotated to identify the target text segments.
  2. Fine-tuning: A pre-trained BERT model is fine-tuned on this labeled dataset.
  3. Classification: The fine-tuned model is used to predict classification scores for sentences in unseen documents.
  4. Identification: Domain knowledge and model scores are combined to infer the precise section of interest.

Key novelties include:

  • Windowing for Training Data Augmentation: To overcome the scarcity of labeled data, ANTS creates overlapping "windows" of tokens from both positive (target) and negative (non-target) sections. This effectively expands the training dataset without requiring more manual annotation.
  • Combined 1-block and 5-block Scores for Prediction: Instead of relying solely on individual sentence scores (1-block), ANTS also considers scores for blocks of five sentences (5-block). This incorporates contextual information from neighboring sentences, improving the accuracy of identifying continuous target sections, especially for shorter or less distinct sentences.
  • Thresholding and Longest Continuous Section: A threshold is applied to the prediction scores, and the longest continuous section of sentences falling below this threshold is identified as the target segment. This approach is tailored to extract a single, contiguous block of text, as required by the specific task.

Datasets

The paper uses a dataset of 393 Form 10-K documents filed by S&P 500 companies from November 2020 through March 2021. The specific task involved extracting the "Human Capital Disclosure" (HCD) section from these filings. This section is characterized by varying titles, sub-section topics, lengths, and locations across documents, making manual extraction challenging. The authors used Amazon's Mechanical Turk (MTurk) to label a portion of these documents for training and evaluation. No new benchmark dataset was introduced; rather, the ANTS framework was applied to an existing, real-world data source.

Key Insights and Ideas

  • Minimal Data is Powerful: The most striking insight is that ANTS can achieve high performance with as few as 5 training documents. This dramatically lowers the barrier to entry for applying advanced NLP to specialized research problems.
  • Windowing is Crucial for Data Scarcity: The windowing technique for training data augmentation is highly effective. It significantly reduces the variance in performance across different random subsamples of training data, making the model more robust and less sensitive to the specific documents chosen for labeling.
  • Context Matters (Even for Short Segments): Using 5-block scores in conjunction with 1-block scores significantly improves recall, indicating that considering sentence context helps in accurately identifying the full extent of the target section. This is particularly useful for capturing sentences that might otherwise be misclassified due to their inherent ambiguity or brevity.
  • "False Positives" Can Be Valuable: The analysis of "false positives" (sentences flagged by the model but outside the strictly defined target section) revealed a high similarity to actual positive sentences. This suggests that ANTS can identify thematically relevant content that might be valuable for researchers even if it falls outside the primary extraction goal, opening avenues for broader analysis.

Key Results

  • The ANTS framework achieved 96% accuracy in identifying the targeted Human Capital Disclosure (HCD) section using only 5 training examples.
  • The windowing method for training data significantly reduced the variability in performance compared to sentence-only training, leading to more stable and predictable results.
  • The combined 1-block and 5-block score approach for section identification resulted in a substantial improvement in the Jaccard index (from 0.43-0.48 for 1-block only to 0.75-0.79 for combined scores), primarily driven by a significant increase in recall.
  • The similarity analysis showed a high correlation (0.88) between actual positive sentences and "false positive" sentences identified by the model, suggesting the model captures relevant contextual information.

Limitations and Future Work

  • Task Specificity: While the framework is general, the current implementation and evaluation are specific to identifying a single, continuous section (HCD). The authors note that for tasks involving multiple or non-contiguous segments (e.g., boilerplate text in charters), the section identification algorithm would need to be tuned.
  • Computational Cost: Using 5-block scores for prediction increases the computational time compared to 1-block scores alone. However, this is presented as a trade-off for improved performance.
  • Future Work: The authors suggest extending ANTS to identify other types of text segments, such as boilerplate language in corporate charters, or to extract or flag relevant text from semi-structured documents (like HTML) and even text transcriptions. They also envision ANTS as a successor to "word list" based methods, enabling the generation of context-aware metrics. The potential to train models on diverse text sources and democratize deep learning for social science research is highlighted.

Summary updated

BibTeX
@inproceedings{chivers2022ants,
  author = {Brian Chivers and Mason P. Jiang and Wonhee Lee and Amy Ng and Natalya I. Rapstine and Alex Storer},
  title = {{ANTS: A Framework for Retrieval of Text Segments in Unstructured Documents}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure