DISA: A Scientific Writing Advisor with Deep Information Structure Analysis

Hen-Hsen Huang, Hsin-Hsi ChenIJCAI 2017 · DOI

TL;DR

This paper introduces DISA, a novel AI writing assistant that goes beyond grammar checks to analyze the information structure of scientific abstracts. By understanding the role of each sentence (e.g., purpose, method, results), DISA offers advice on improving the abstract's organization and retrieves relevant prior work, aiming to help researchers build stronger arguments and find better references.

What Problem Does This Paper Address?

Writing effective scientific papers is a complex skill that involves not just clear language but also a logical flow of information. While many writing assistant tools focus on surface-level issues like grammar and spelling, they often fail to address the higher-level challenge of structuring an argument. A poorly structured abstract, even if grammatically perfect, can fail to convey its core message or attract the right readers. This paper tackles this gap by developing a system that analyzes and advises on the information structure of scientific abstracts, a crucial component for effective scientific communication.

Methods

DISA employs a multi-stage approach:

  • Preprocessing: The system first segments abstracts into sentences and performs standard Natural Language Processing (NLP) tasks like tokenization, part-of-speech tagging, and dependency parsing.
  • Information Structure Identification: This is the core novelty. DISA uses a linguistically-informed neural network model. This model leverages handcrafted linguistic features (like sentence position, n-grams, POS tags, and dependency relations) alongside the raw text to classify each sentence into predefined categories (e.g., Background, Purpose, Method, Results, Conclusion). A Viterbi algorithm is used for sequential modeling, recognizing that sentence roles often follow a typical order.
  • Knowledge Retrieval: Once the information structure is identified, DISA focuses on the "Purpose" sentences to understand the research goals. It then uses a GRU-based autoencoder to represent sentences as dense vectors, capturing semantic similarity. This autoencoder, combined with TF-IDF weights, forms a hybrid retrieval model. It retrieves relevant passages from a collection of pre-analyzed scientific papers, specifically focusing on related methods and background information for the identified research goals.
  • Topic Modeling (LDA): To further refine retrieval results, a Latent Dirichlet Allocation (LDA) model is used to cluster abstracts into topics. DISA infers the topic of the user's abstract and filters retrieved results to those within the same topic, ensuring relevance.

Datasets

  • NTHU Academic Writing Database: This dataset, containing 210,771 sentences from 4,380 EECS papers, was used for training and evaluation. It includes abstracts annotated with information structure labels by linguistic experts. Table 1 in the paper details the categories and counts of annotated sentences.

Key Insights and Ideas

  • Information Structure is Key: The paper strongly emphasizes that the arrangement of information in an abstract is as critical as its grammatical correctness for conveying its message effectively.
  • Beyond Surface-Level Analysis: DISA represents a shift from traditional grammar checkers to a deeper semantic and structural analysis of scientific text.
  • Neural Networks for Structure: The use of linguistically-informed neural networks for information structure identification demonstrates a powerful way to combine linguistic expertise with machine learning for complex NLP tasks.
  • Autoencoders for Semantic Retrieval: GRU-based autoencoders are shown to be effective in capturing sentence semantics for retrieving relevant prior work, going beyond simple keyword matching.
  • Hybrid Retrieval for Relevance: Combining autoencoder-based semantic retrieval with topic modeling (LDA) provides a more robust and contextually relevant knowledge retrieval system.

Key Results

  • Information Structure Identification: The linguistically-informed neural network model achieved a macro-average f-score of 75.04% in a five-way classification task on the NTHU dataset, demonstrating good performance in identifying sentence roles.
  • Advice Generation: The system can identify structural anomalies (e.g., missing "Purpose" or "Results" statements) by comparing the abstract's structure distribution to a norm derived from the dataset. It then provides specific advice to the user.
  • Knowledge Retrieval: The system successfully retrieves related methods and background information relevant to the identified research goals, filtered by topic, suggesting its utility for literature review and argument building.

Limitations and Future Work

  • Dataset Size and Diversity: The current information structure detector is trained on a moderately sized EECS dataset. The capability of knowledge retrieval is also limited by the size and diversity of the backend database.
  • Future Work: The authors plan to incrementally import more scientific articles from the public domain to expand the database. They also intend to conduct user studies to quantitatively measure the system's performance in real-world scenarios. The sequential modeling component can be optionally disabled for abstracts with extremely poor structure.

Summary updated

BibTeX
@inproceedings{huang2017disa,
  author = {Hen-Hsen Huang and Hsin-Hsi Chen},
  title = {{DISA: A Scientific Writing Advisor with Deep Information Structure Analysis}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/773},
}
Figure