Abstract
Reading and writing research papers is one of the most privileged abilities that a qualified researcher should master. However, it is difficult for new researchers (e.g., students) to fully grasp this ability. It would be fascinating if we could train an intelligent agent to help people read and summarize papers, and perhaps even discover and exploit the potential knowledge clues to write novel papers. Although there have been existing works focusing on summarizing (i.e., reading) the knowledge in a given text or generating (i.e., writing) a text based on the given knowledge, the ability of simultaneously reading and writing is still under development. Typically, this requires an agent to fully understand the knowledge from the given text materials and generate correct and fluent novel paragraphs, which is very challenging in practice. In this paper, we propose a Deep ReAder-Writer (DRAW) network, which consists of a Reader that can extract knowledge graphs (KGs) from input paragraphs and discover potential knowledge, a graph-to-text Writer that generates a novel paragraph, and a Reviewer that reviews the generated paragraph from three different aspects. Extensive experiments show that our DRAW network outperforms considered baselines and several state-of-the-art methods on AGENDA and M-AGENDA datasets.
Introduction
The exponential growth of scientific literature presents a significant challenge for researchers. With thousands of papers published daily, staying abreast of a field is nearly impossible. A 2019 study highlighted that the average scientist reads only about 264 papers annually, a mere fraction of the total output. To address this information overload, researchers from the South China University of Technology and the University of Adelaide have introduced the Deep ReAder-Writer (DRAW) network. Unlike traditional tools that focus solely on summarization or text generation, DRAW is designed to synthesize new insights by simultaneously reading multiple sources and writing coherent, novel research paragraphs.
The DRAW Framework
The DRAW network is an end-to-end framework comprising three specialized modules: the Reader, the Writer, and the Reviewer. This architecture mimics the human research process of gathering information, synthesizing it, and refining the output based on feedback.
1. The Reader: Text-to-Graph Extraction
The Reader module utilizes the SciIE system to extract entities and relations from input abstracts. A key innovation here is the use of link prediction. By identifying "potential knowledge"—connections between entities that are not explicitly stated but can be inferred—the Reader constructs an enriched Knowledge Graph (KG). This allows the agent to "connect the dots" across disparate research papers.
2. The Writer: Graph-to-Text Generation
The Writer module employs a dual-encoding strategy to translate the enriched KG into natural language. It uses a global graph encoder (Transformer-based) to capture the overall topology and context, alongside a local graph encoder (Graph Attention Network) to focus on immediate entity neighborhoods. A text decoder then synthesizes these representations into a fluent paragraph.
3. The Reviewer: Reinforcement Learning Feedback
The Reviewer acts as an automated "advisor," providing multi-dimensional feedback to refine the Writer's output via reinforcement learning. It evaluates the generated text based on three rewards:
- Quality Reward: Measures fluency using standard metrics like BLEU.
- Adversarial Reward: A discriminator trained to distinguish between human and machine-generated text, pushing the model toward more natural phrasing.
- Alignment Reward: Ensures the generated text remains factually consistent with the source Knowledge Graph.
Experimental Evaluation
The authors evaluated DRAW on two primary benchmarks: AGENDA, containing 40,000 abstracts and KGs from AI proceedings, and M-AGENDA, a multi-document dataset created by clustering related abstracts to test synthesis capabilities.
Quantitative Results
DRAW demonstrated superior performance over state-of-the-art models. On the AGENDA dataset, it achieved a BLEU score of 19.60, significantly higher than the previous best of 18.01 (CGE-LW).
Human Evaluation and Turing Test
In human evaluations, DRAW consistently scored higher in grammar, coherence, and informativeness compared to baselines like PaperRobot. Furthermore, in a "Turing Test" setup, human evaluators mistook DRAW-generated paragraphs for human-written ones 48% of the time. For context, humans only correctly identified human-written abstracts 68% of the time, suggesting that DRAW's output is approaching human-level quality in specific contexts.
Conclusion and Future Work
The DRAW network represents a significant step toward an automated research assistant capable of synthesizing scientific knowledge. By integrating link prediction and a multi-faceted Reviewer module, the system overcomes the limitations of simple word-matching models. While currently limited to single-paragraph generation, future work aims to extend DRAW to full-length documents and the discovery of entirely new scientific entities.
Summary updated