Novelty Goes Deep. A Deep Neural Solution To Document Level Novelty Detection

Tirthankar Ghosal, Vignesh Edithal, Asif Ekbal, Pushpak Bhattacharyya, George Tsatsaronis, Srinivasa Satya Sameer Kumar ChivukulaCOLING 2018

TL;DR

This paper introduces a novel deep learning approach, RDV-CNN, for document-level novelty detection. By learning to represent target documents relative to source documents, their method significantly outperforms existing techniques, offering a more robust way to identify genuinely new information in the ever-growing sea of digital text.

What Problem Does This Paper Address?

The internet is awash with information, and a significant portion of it is redundant. Identifying genuinely novel documents is crucial for applications like information retrieval, summarization, and even plagiarism detection. While sentence-level novelty detection has seen some progress, detecting novelty at the document level, which requires understanding deeper semantic relationships and context, remains a significant challenge. Existing methods often rely on hand-crafted features or struggle to capture the nuanced semantic differences that define novelty. This paper aims to bridge this gap by proposing a deep neural network that can learn to distinguish novel from non-novel documents without manual feature engineering.

Methods

The core of the proposed solution is the Relative Document Vector (RDV), a novel representation that captures the relationship between a target document and its source documents. Here's how it works:

  1. Sentence Encoding: A pre-trained bi-directional LSTM (Long Short-Term Memory) network, trained on the SNLI dataset (Stanford Natural Language Inference), is used to generate universal sentence embeddings. This encoder is designed to capture rich semantic relationships between sentence pairs.
  2. Comparator Module: For each sentence in the target document, this module finds the most similar sentence in the source document(s) based on cosine similarity of their embeddings.
  3. Aggregator Module (Relative Sentence Vector - RSV): This module constructs a feature vector for each target sentence by concatenating its embedding with the embedding of its closest source sentence, along with their absolute element-wise difference and element-wise product. This RSV aims to capture both "closeness" (for non-novel sentences) and "diversity" (for novel sentences).
  4. Relative Document Vector (RDV): The RSVs for all sentences in a target document are aggregated to form a document matrix. This matrix serves as the input to the final classification module.
  5. CNN Module: A Convolutional Neural Network (CNN) is trained on these RDVs to classify the target document as either novel or non-novel with respect to the source documents. The CNN is chosen for its ability to automatically extract features from these representations.

Datasets

The authors evaluated their approach on three datasets:

  • APWSJ Corpus: A news dataset comprising articles from Associated Press and Wall Street Journal, designed for document-level novelty detection.
  • Webis Crowd Paraphrase Corpus (Webis CPC): Used to simulate non-novelty detection by treating paraphrased documents as non-novel.
  • TAP-DLND 1.0: A recently published, balanced dataset specifically for document-level novelty detection, which the authors highlight as an ideal candidate for their experiments due to its semantic nature.

Key Insights and Ideas

  • Semantic Relativity is Key: The paper strongly emphasizes that novelty is not an absolute property but is defined relative to existing knowledge. Their RDV representation directly models this by explicitly comparing target sentences to source sentences.
  • Deep Learning for Nuance: The authors posit that deep neural networks are well-suited to capture the subtle semantic and contextual cues required for document-level novelty detection, moving beyond simpler lexical or statistical methods.
  • Paraphrase Detection as a Proxy: Using paraphrase detection as a simulation for non-novelty detection on the Webis CPC dataset provides a valuable way to test the model's ability to distinguish semantically similar but informationally redundant content.
  • RDV-CNN Architecture: The combination of learning relative sentence representations (RSV) and then using a CNN for document classification proves effective, allowing the network to learn complex interactions.

Key Results

The RDV-CNN model demonstrated strong performance across all datasets:

  • APWSJ: Achieved a 22.9% mistake rate, outperforming previous methods by a significant margin.
  • Webis-CPC: Achieved an accuracy of 78.02%, surpassing various baselines and established novelty detection measures.
  • TAP-DLND 1.0: Achieved an accuracy of 84.53%, significantly outperforming existing state-of-the-art methods and baselines by up to 12% in accuracy. The paper-values for F1 score were also statistically significant.

Limitations and Future Work

  • Multiple Premises: The current RDV-CNN can only handle one source premise sentence per target sentence. The authors intend to accommodate multiple premises in future work, as errors often arise when a target sentence relates to several source sentences.
  • Computational Cost: With a large number of source documents, the architecture can become computationally expensive. The authors plan to mitigate this.
  • Relevance Judgment: Incorporating explicit relevance judgment as a component of the architecture is suggested for future improvement.
  • Attention Mechanism: Investigating the role of attention mechanisms in creating the RDV is another promising avenue for future research.
  • Temporality: The current model reserves the exploration of temporality (how novelty changes over time) for future work.

Summary updated

BibTeX
@inproceedings{ghosal2018novelty,
  author = {Tirthankar Ghosal and Vignesh Edithal and Asif Ekbal and Pushpak Bhattacharyya and George Tsatsaronis and Srinivasa Satya Sameer Kumar Chivukula},
  title = {{Novelty Goes Deep. A Deep Neural Solution To Document Level Novelty Detection}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure