Exploring the Ideal Depth of Neural Network when Predicting Question Deletion on Community Question Answering

Souvick Ghosh, Satanu GhosharXiv 2019 · arXiv

TL;DR

This paper investigates the optimal depth of neural networks for predicting question deletion in Community Question Answering (CQA) forums. The researchers found that while moderate depth (2–10 hidden layers) achieves high accuracy (over 90%), excessively deep models suffer from vanishing gradients, leading to a significant drop in performance and increased computational costs.

What Problem Does This Paper Address?

Community Question Answering (CQA) platforms like Stack Overflow, Quora, and Brainly are often overwhelmed by a high volume of posts. To maintain quality, moderators must identify and delete low-quality, vague, or inappropriate questions. Manually filtering millions of posts is unsustainable, necessitating automated prediction models.

While deep learning has seen success in other fields, its application in Neural Information Retrieval (IR) is still exploratory. The authors tackle a fundamental architectural question: How does the depth of a neural network—specifically the number of hidden layers—impact the accuracy and efficiency of predicting question deletion?

Methods

The authors employed a supervised learning approach using a Multilayer Perceptron (MLP) architecture.

  • Data Representation: To process text, they used Google’s pre-trained Word2Vec model (300-dimensional embeddings). For each question, word embeddings were concatenated and zero-padded to create a fixed-length input vector of 72,001 dimensions.
  • Architecture: They systematically varied the number of hidden layers from 1 to 100. The layers were densely connected, using ReLU activation functions in hidden layers and a Sigmoid function for the final binary classification (deleted vs. not deleted).
  • Training Details: The models included a 5% dropout rate to prevent overfitting and were trained for 150 epochs using binary cross-entropy as the loss function.
Figure 1: Deep Neural Model

Datasets

The study utilized a dataset of 6,000 questions collected from Brainly, a popular educational CQA site.

  • Balance: The dataset was perfectly balanced, with 3,000 deleted questions and 3,000 non-deleted questions.
  • Split: 5,000 questions were used for training and 1,000 for testing. 10% of the training data was reserved for validation in each iteration.

Key Insights and Ideas

  • The "Sweet Spot" of Depth: Accuracy does not scale infinitely with depth. The model's performance improved significantly as it moved from 1 to 3 layers, but then plateaued. The best results were consistently found between 2 and 10 hidden layers.
  • The Vanishing Gradient Trap: Beyond 25 layers, accuracy began to decline. At 50 and 100 layers, the model's performance plummeted to roughly 50%—no better than random guessing. The authors attribute this to the vanishing gradient problem, where the error signal becomes too small to effectively update weights in very deep architectures during backpropagation.
  • Linear Computational Cost: While accuracy eventually drops, the training time increases almost linearly with the number of layers. This highlights a clear trade-off: adding layers beyond the "sweet spot" costs more time while yielding worse results.

Key Results

The quantitative findings, summarized in the paper's Table 1 and Figure 2, provide a clear picture of the depth-performance relationship:

  • Peak Performance: The highest test accuracy reached 97.7% with 10 hidden layers.
  • Efficiency: Models with 2 to 5 layers offered the best balance, achieving >90% accuracy with significantly lower training times (~6,450 to 6,644 seconds) compared to deeper models.
  • Failure at Depth: By 100 layers, test accuracy fell to 49.2%, while training time surged to 15,644 seconds.
Graphs showing how training time increases linearly with depth, while accuracy peaks early and then drops sharply.

Limitations and Future Work

The authors acknowledge that there is no "one-size-fits-all" architecture for Neural IR. The findings are specific to the MLP architecture and the Brainly dataset.

Future directions include: * Investigating other neural architectures (like CNNs or Transformers) for the same task. * Exploring how other hyperparameters (beyond just depth) can be tuned to push accuracy further once the depth-saturation point is reached. * Applying these insights to develop more scalable and accurate architectures for broader information retrieval problems.

Summary updated

Abstract

In recent years, Community Question Answering (CQA) has emerged as a popular platform for knowledge curation and archival. An interesting aspect of question answering is that it combines aspects from natural language processing, information retrieval, and machine learning. In this paper, we have explored how the depth of the neural network influences the accuracy of prediction of deleted questions in question-answering forums. We have used different shallow and deep models for prediction and analyzed the relationships between number of hidden layers, accuracy, and computational time. The results suggest that while deep networks perform better than shallow networks in modeling complex non-linear functions, increasing the depth may not always produce desired results. We observe that the performance of the deep neural network suffers significantly due to vanishing gradients when large number of hidden layers are present. Constantly increasing the depth of the model increases accuracy initially, after which the accuracy plateaus, and finally drops. Adding each layer is also expensive in terms of the time required to train the model. This research is situated in the domain of neural information retrieval and contributes towards building a theory on how deep neural networks can be efficiently and accurately used for predicting question deletion. We predict deleted questions with more than 90\% accuracy using two to ten hidden layers, with less accurate results for shallower and deeper architectures.
BibTeX
@misc{ghosh2019exploring,
  author = {Souvick Ghosh and Satanu Ghosh},
  title = {{Exploring the Ideal Depth of Neural Network when Predicting Question Deletion on Community Question Answering}},
  year = {2019},
  eprint = {1912.03585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.03585},
}
Figure