aiai at the FinSBD-2 Task: Sentence, list and Item Boundary Detection and Items classification of Financial Texts Using Data Augmentation and Attention

Ke Tian, Hua Chen, Jie YangCOLING 2020

TL;DR

This paper tackles the challenging task of accurately identifying sentence, list, and item boundaries within noisy financial documents, a crucial step for information extraction. The authors propose a deep attention model combined with data augmentation techniques to overcome the limitations of traditional methods when dealing with unstructured and error-prone financial text.

What Problem Does This Paper Address?

Extracting structured information from financial documents, particularly those originating from PDFs, is a significant challenge. These documents often contain "noisy" text due to OCR errors, inconsistent formatting, and the presence of visual cues like bullet points and section hierarchies that don't always align with standard linguistic structures. Traditional sentence boundary detection (SBD) methods, which perform well on clean, formal text, struggle with this noise. The FinSBD-2 task specifically addresses this by requiring not only sentence boundary detection but also the identification of list and item boundaries, and the classification of these items into predefined categories. This is essential for making financial information more accessible and processable for downstream tasks.

Methods

The core of the proposed method lies in a deep attention model augmented with a novel data augmentation strategy.

  1. Data Augmentation: To enrich the limited training data and capture richer contextual information, the authors augment each tokenized word. This augmentation involves concatenating the token with its preceding and succeeding n tokens, along with spatial and semantic information derived from the text's layout (e.g., coordinates, width, and height of the first character). This creates a fixed-length representation for each word, effectively turning the boundary detection problem into a classification task for these augmented "sentences."
  2. Deep Attention Model: An attention-based LSTM (Long Short-Term Memory) network is employed. The LSTM processes the sequence of augmented word representations. The attention mechanism allows the model to dynamically focus on the most relevant parts of the input sequence when making predictions. This is particularly useful for identifying sentence boundaries, where context from surrounding words is crucial.
  3. Two-Stage Approach: The system operates in two main stages:
  4. Boundary Detection: The augmented word representations are classified into four types: "OS" (other sentence), "BS" (beginning of sentence/list/item), "ES" (end of sentence/list/item), and "IS" (independent sentence/list/item). This classification is then used to infer the start and end character indices of sentences, lists, and items.
  5. Sentence Classification: The detected sentence texts are then fed into another deep attention model to classify them into four specific item types (item1, item2, item3, item4).
  6. Ensemble: For the final prediction, a 5-fold cross-validation approach is used, and the mean of the predicted probabilities from these folds is taken to produce the final result.

Datasets

The paper utilizes the FinSBD2-2020 shared task dataset, which consists of noisy English and French financial texts extracted from PDF documents. The dataset includes:

  • Training Data: 29 labeled JSON files (6 English, 23 French).
  • Test Data: 12 prediction JSON files (2 English, 10 French).

The data is annotated with sentence, list, and item boundaries, as well as item classifications.

Key Insights and Ideas

  • Spatial and Semantic Information is Crucial: The paper highlights that simply relying on textual content is insufficient for noisy financial documents. Incorporating spatial layout information (coordinates, dimensions) and contextual word sequences significantly improves the ability to detect boundaries.
  • Data Augmentation for Limited Data: The proposed data augmentation strategy is key to overcoming the challenge of limited training data. By creating richer, fixed-length representations, the model can learn more effectively.
  • Attention for Contextual Understanding: The use of attention mechanisms in the LSTM architecture is vital for capturing long-range dependencies and subtle contextual cues that signal sentence or item boundaries, especially in the presence of noise.
  • Unified Approach for Boundary Detection and Classification: The paper demonstrates a successful approach to handle both boundary detection and item classification within a single framework, leveraging similar deep learning techniques.

Key Results

The proposed method achieved competitive results on the FinSBD2 task.

  • English Task: The proposed method achieved an F-score of 0.413 for Subtask 1 (boundary detection) and 0.35 for Subtask 2 (item classification).
  • French Task: The proposed method achieved an F-score of 0.471 for Subtask 1 and 0.35 for Subtask 2.

While these scores are respectable and placed the "aiai" team on the final leaderboard, the authors acknowledge that performance still needs improvement compared to an "ideal score."

Limitations and Future Work

The authors identify several limitations and suggest future research directions:

  • Preprocessing: The preprocessing of text might not be sufficiently robust for all cases.
  • Parameter Tuning: Limited time due to busy schedules prevented extensive parameter tuning.
  • Model Exploration: Other advanced models like BERT (Bidirectional Encoder Representations from Transformers) and Named Entity Recognition (NER) methods were not explored but could potentially improve performance.
  • Data Size: The limited amount of training data may have influenced the model's performance.
  • Word-Char Mismatch: In the English data, some words might not perfectly align with their relative character indices.

Future work includes exploring more parameter tuning, investigating alternative models like BERT, and potentially expanding the dataset or employing more advanced data augmentation techniques.

Summary updated

BibTeX
@inproceedings{tian2020aiai,
  author = {Ke Tian and Hua Chen and Jie Yang},
  title = {{aiai at the FinSBD-2 Task: Sentence, list and Item Boundary Detection and Items classification of Financial Texts Using Data Augmentation and Attention}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure