Prediction Algorithm for Heat Demand of Science and Technology Topics Based on Time Convolution Network

Haiyan Cui, Yawen Li, Xin XuarXiv 2022 · OpenReview

Abstract

Thanks to the rapid development of deep learning, big data analysis technology is not only widely used in the field of natural language processing, but also more mature in the field of numerical prediction. It is of great significance for the subject heat prediction and analysis of science and technology demand data. How to apply theme features to accurately predict the theme heat of science and technology demand is the core to solve this problem. In this paper, a prediction method of subject heat of science and technology demand based on time convolution network (TCN) is proposed to obtain the subject feature representation of science and technology demand. Time series prediction is carried out based on TCN network and self-attention mechanism, which increases the accuracy of subject heat prediction of science and technology demand data. Experiments show that the prediction accuracy of this algorithm is better than other time series prediction methods on the real science and technology demand datasets.

Introduction

In the rapidly evolving landscape of Research and Development (R&D), identifying which technology topics are gaining traction is crucial for strategic decision-making. "Topic heat"—the intensity of interest in a specific subject over time—serves as a vital metric for researchers and policymakers. However, predicting this heat is challenging due to the inherent complexity of science and technology (S&T) demand data, which often exhibits non-linear trends, seasonality, and random fluctuations.

Trends in word frequency for various technology topics from 2015 to 2020, illustrating the complexity of topic heat prediction.

Traditional models like ARIMA struggle with non-stationary data, while standard Recurrent Neural Networks (RNNs) often fail to capture long-range dependencies effectively. This paper proposes the SHDP-TCN framework to address these limitations by integrating thematic features with advanced deep learning architectures.

Methodology: The SHDP-TCN Framework

The proposed SHDP-TCN (Science and Technology Heat Demand Prediction - TCN) model utilizes three core components to enhance prediction accuracy:

  1. Self-Attention Mechanism: Acting as a pre-processor, this layer weights historical time steps based on their relevance to the current prediction, highlighting local correlations within the sequence.
  2. Temporal Convolutional Network (TCN): By employing causal and dilated convolutions, the TCN achieves a large receptive field, allowing it to process long historical sequences in parallel without information leakage from the future.
  3. Thematic Feature Integration: The model incorporates semantic category data (topic word features) to provide industry-specific context, strengthening the predictive power beyond simple numerical trends.
Architecture of the SHDP-TCN model, featuring the self-attention mechanism and TCN residual blocks.

Experimental Setup and Results

The study utilized a dataset of 30,872 technology demand records spanning from 2012 to 2021, aggregated at monthly intervals. The data was categorized into 10 industry sectors, including Manufacturing, Agriculture, and Electronic Information. The model was trained on data prior to June 2019 and tested on the subsequent period.

Comparative analysis against baseline models demonstrates the superiority of SHDP-TCN, particularly when thematic features are included:

Model Precision Recall F1-Score
ARIMA 0.4738 0.3036 0.3701
LSTM 0.5601 0.5064 0.5319
CNN 0.5417 0.5127 0.5268
TCN (Standard) 0.6125 0.4792 0.5377
SHDP-TCN (Ours) 0.7002 0.6903 0.6952

The SHDP-TCN achieved a precision of 0.7002, significantly outperforming traditional time-series methods and standard deep learning architectures. The results highlight that the "what" of a technology (thematic context) is just as important as the "when" (historical trend).

Conclusion

The integration of self-attention and thematic features within a TCN framework provides a robust solution for predicting the heat of science and technology topics. Future research will focus on increasing data granularity and incorporating external variables, such as policy shifts and economic indicators, to further refine prediction accuracy.

Summary updated

Abstract

Thanks to the rapid development of deep learning, big data analysis technology is not only widely used in the field of natural language processing, but also more mature in the field of numerical prediction. It is of great significance for the subject heat prediction and analysis of science and technology demand data. How to apply theme features to accurately predict the theme heat of science and technology demand is the core to solve this problem. In this paper, a prediction method of subject heat of science and technology demand based on time convolution network (TCN) is proposed to obtain the subject feature representation of science and technology demand. Time series prediction is carried out based on TCN network and self attention mechanism, which increases the accuracy of subject heat prediction of science and technology demand data Experiments show that the prediction accuracy of this algorithm is better than other time series prediction methods on the real science and technology demand datasets.
BibTeX
@misc{cui2022prediction,
  author = {Haiyan Cui and Yawen Li and Xin Xu},
  title = {{Prediction Algorithm for Heat Demand of Science and Technology Topics Based on Time Convolution Network}},
  year = {2022},
  eprint = {2203.10718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.10718},
}
Figure