TL;DR
This paper tackles the challenge of summarizing lengthy financial reports by developing an extractive summarization system. They leverage Determinantal Point Processes (DPPs) to select high-quality and diverse sentences, focusing on the "narrative" sections of reports to create more informative summaries.
What Problem Does This Paper Address?
Financial annual reports are notoriously long and dense, often exceeding 250 pages. For investors and stakeholders, sifting through this information to extract key insights is a time-consuming and difficult task. Existing methods often struggle with the sheer volume and potential redundancy within these documents. This paper aims to automate the process of generating concise summaries that capture the essential narrative information from these reports, making them more accessible and useful.
Methods
The authors propose an extractive summarization system based on Determinantal Point Processes (DPPs). The core idea is to treat the document as a set of sentences and use DPPs to sample a subset of sentences that are both high-quality and diverse.
The process involves several key steps:
- Pre-processing: This is a crucial step for financial reports. The system first identifies and extracts "narrative sections" (e.g., "Highlights," "Chairman's statement") and filters out "back-end" sections containing financial statements, tables, and numbers. They also clean the text by removing headers, footers, and non-narrative content.
- Feature Selection: To represent the document and guide sentence selection, they construct a kernel matrix
L. This matrix incorporates three statistical features for each sentence: - Sentence Length (SL): Measures the length of a sentence.
- Sentence Position (SP): Indicates the position of a sentence within the document.
-
Sentence Coverage (SC): Quantifies how many sentences a word within a given sentence appears in. The quality of a sentence (
qi) is derived from these features, and the similarity between sentences (Simij) is calculated using Jaccard similarity. The kernel matrixLthen combines quality and similarity to represent sentence relationships. -
Sentence Sampling: Using the constructed kernel matrix
L, they apply a discrete DPPs sampling algorithm. This algorithm selects sentences by projecting them into a lower-dimensional space where vector magnitude indicates importance and cosine similarity reflects sentence relatedness. The process iteratively selects high-quality sentences while ensuring diversity by removing feature vectors that contribute to the selected sentence's magnitude, thus promoting low similarity to previously chosen sentences.
Datasets
The system was developed and evaluated using the FNS-Summarisation 2020 shared task dataset. This dataset consists of 3863 UK annual reports from firms listed on The London Stock Exchange (LSE), covering the period between 2002 and 2017. The reports are lengthy, averaging around 80 pages but sometimes exceeding 250. The dataset is divided into training, validation, and testing sets, with gold-standard summaries provided for training and validation. For the testing dataset, participants only had access to the full texts.
Key Insights and Ideas
- Narrative Focus is Key: The paper highlights that focusing on specific "narrative" sections of financial reports is more effective for summarization than processing the entire document. This is because these sections contain the qualitative insights and management commentary that are most valuable for a summary.
- Sentence Coverage as a Strong Feature: Among the tested features (SL, SP, SC), Sentence Coverage (SC) emerged as the most effective for building the kernel matrix. This suggests that understanding how widely a sentence's words are distributed across the document is a strong indicator of its importance and relevance.
- DPPs for Quality and Diversity: The use of Determinantal Point Processes is a well-suited approach for extractive summarization, as it inherently balances the selection of important (high-quality) sentences with the need for diversity, preventing redundancy and ensuring a broader coverage of topics within the summary.
Key Results
The authors submitted three versions of their system to the FNS-Summarisation 2020 shared task, varying the features used for the kernel matrix:
- CIST-BUPT-RUN1: Merged all three features (SC, SL, SP).
- CIST-BUPT-RUN2: Used Sentence Length (SL).
- CIST-BUPT-RUN3: Used Sentence Coverage (SC).
On the official validation set, CIST-BUPT-RUN3 (using SC) achieved the best performance with ROUGE-1/F of 0.313, ROUGE-2/F of 0.079, and ROUGE-L/F of 0.167.
On the testing set, CIST-BUPT-RUN3 also performed best across various ROUGE metrics. For instance, its R-L/F score was 0.329, and R-1/F was 0.428. The paper notes that ROUGE scores on the testing set were higher than on the training set, which they attribute to the evaluation methodology involving merging multiple golden abstracts.
Limitations and Future Work
- Computational Resources: The authors mention that due to the COVID-19 pandemic, limited server availability prevented them from training more advanced models like Sentence2Vec, which they believe could further improve performance.
- Future Work: They suggest exploring other document representation methods, such as training a Sentence2Vec model using neural networks, as a promising direction for future research to enhance summary quality. They also plan to investigate other feasible methods for document representation.
Summary updated