TL;DR
This paper investigates the optimal depth of neural networks for predicting question deletion in Community Question Answering (CQA) forums. The researchers found that while moderate depth (2–10 hidden layers) achieves high accuracy (over 90%), excessively deep models suffer from vanishing gradients, leading to a significant drop in performance and increased computational costs.
What Problem Does This Paper Address?
Community Question Answering (CQA) platforms like Stack Overflow, Quora, and Brainly are often overwhelmed by a high volume of posts. To maintain quality, moderators must identify and delete low-quality, vague, or inappropriate questions. Manually filtering millions of posts is unsustainable, necessitating automated prediction models.
While deep learning has seen success in other fields, its application in Neural Information Retrieval (IR) is still exploratory. The authors tackle a fundamental architectural question: How does the depth of a neural network—specifically the number of hidden layers—impact the accuracy and efficiency of predicting question deletion?
Methods
The authors employed a supervised learning approach using a Multilayer Perceptron (MLP) architecture.
- Data Representation: To process text, they used Google’s pre-trained Word2Vec model (300-dimensional embeddings). For each question, word embeddings were concatenated and zero-padded to create a fixed-length input vector of 72,001 dimensions.
- Architecture: They systematically varied the number of hidden layers from 1 to 100. The layers were densely connected, using ReLU activation functions in hidden layers and a Sigmoid function for the final binary classification (deleted vs. not deleted).
- Training Details: The models included a 5% dropout rate to prevent overfitting and were trained for 150 epochs using binary cross-entropy as the loss function.
Datasets
The study utilized a dataset of 6,000 questions collected from Brainly, a popular educational CQA site.
- Balance: The dataset was perfectly balanced, with 3,000 deleted questions and 3,000 non-deleted questions.
- Split: 5,000 questions were used for training and 1,000 for testing. 10% of the training data was reserved for validation in each iteration.
Key Insights and Ideas
- The "Sweet Spot" of Depth: Accuracy does not scale infinitely with depth. The model's performance improved significantly as it moved from 1 to 3 layers, but then plateaued. The best results were consistently found between 2 and 10 hidden layers.
- The Vanishing Gradient Trap: Beyond 25 layers, accuracy began to decline. At 50 and 100 layers, the model's performance plummeted to roughly 50%—no better than random guessing. The authors attribute this to the vanishing gradient problem, where the error signal becomes too small to effectively update weights in very deep architectures during backpropagation.
- Linear Computational Cost: While accuracy eventually drops, the training time increases almost linearly with the number of layers. This highlights a clear trade-off: adding layers beyond the "sweet spot" costs more time while yielding worse results.
Key Results
The quantitative findings, summarized in the paper's Table 1 and Figure 2, provide a clear picture of the depth-performance relationship:
- Peak Performance: The highest test accuracy reached 97.7% with 10 hidden layers.
- Efficiency: Models with 2 to 5 layers offered the best balance, achieving >90% accuracy with significantly lower training times (~6,450 to 6,644 seconds) compared to deeper models.
- Failure at Depth: By 100 layers, test accuracy fell to 49.2%, while training time surged to 15,644 seconds.
Limitations and Future Work
The authors acknowledge that there is no "one-size-fits-all" architecture for Neural IR. The findings are specific to the MLP architecture and the Brainly dataset.
Future directions include: * Investigating other neural architectures (like CNNs or Transformers) for the same task. * Exploring how other hyperparameters (beyond just depth) can be tuned to push accuracy further once the depth-saturation point is reached. * Applying these insights to develop more scalable and accurate architectures for broader information retrieval problems.
Summary updated