Earth Mover's Distance Pooling over Siamese LSTMs for Automatic Short Answer Grading

Sachin Kumar, Soumen Chakrabarti, Shourya Roy

Abstract

This paper introduces a novel neural architecture for Automatic Short Answer Grading (ASAG) that treats grading as a "transportation" problem. By combining Siamese bi-LSTMs with Earth Mover’s Distance (EMD) pooling, the system effectively matches concepts between student and model answers, significantly outperforming standard attention mechanisms and traditional regression baselines on computer science and science datasets.

The ASAG Challenge

Grading short, open-ended student answers is a significant bottleneck in education. Unlike multiple-choice questions, short answers (ranging from a few words to a few sentences) exhibit high linguistic variation—students can express the same correct concept in many different ways. The authors identify three core challenges in ASAG that distinguish it from standard Natural Language Inference (NLI):

  1. Asymmetry: A student answer doesn't need to be a perfect paraphrase; it just needs to cover the key concepts of the model answer. Extra information is usually acceptable.
  2. Ordinality: Grades are not binary; they exist on a scale (e.g., 0 to 5).
  3. Data Scarcity: High-quality, human-graded ASAG datasets are relatively small compared to the massive corpora used in other NLP tasks.

Proposed Architecture

The authors propose a three-stage neural pipeline designed to address these challenges:

  1. Siamese bi-LSTMs: The model answer and student answer are processed by identical bidirectional LSTMs with shared weights. This generates a sequence of hidden state vectors for both answers, capturing contextual meaning.
  2. EMD Pooling Layer: This is the paper's primary innovation. Instead of using the final LSTM state or standard attention, they compute the Earth Mover’s Distance (EMD)—also known as the Wasserstein distance—between the sets of hidden states. EMD finds the "minimum cost" to transform the distribution of meanings in the student answer into the model answer. To maintain efficiency, they use the Sinkhorn-Knopp algorithm for a faster, differentiable $O(d^2)$ approximation.
  3. SVOR Output Layer: To handle the ordinal nature of grades, they use Support Vector Ordinal Regression (SVOR) with a "tub loss" (a two-sided hinge loss). This is more effective than standard classification or simple linear regression.
The LSTM-EMD-SVOR architecture diagram showing the Siamese LSTM encoders followed by the EMD pooling layer and regression output.

To mitigate data scarcity, the authors employed a Data Augmentation strategy, assuming that student answers receiving perfect scores are functionally equivalent to the instructor's model answer, thereby creating thousands of new training pairs.

Experimental Setup

The model was evaluated on two major benchmarks:

  • Mohler CS Dataset: 2,273 student responses to 80 undergraduate Data Structures questions, graded on a 0–5 scale.
  • SemEval-2013 (SRA Task): Specifically the ScientsBank subset, containing ~10,000 answers across 15 science domains. Categorical labels were mapped to an ordinal scale (2, 1, 0) for evaluation.

Key Insights and Results

Global Matching vs. Local Attention

Surprisingly, standard attention mechanisms performed poorly. The authors argue that EMD provides a more robust, global alignment of concepts across the entire sequence compared to the "fickle" nature of token-level attention in short-answer contexts. Furthermore, computing EMD over LSTM hidden states proved far superior to using raw word embeddings, as the states capture long-distance semantics and context.

Visualizing "Flow"

EMD heatmaps (transportation matrices) provide a form of interpretability, showing exactly which parts of a student's answer "fill the bucket" of the model answer's required concepts.

An EMD transportation matrix visualizing the semantic alignment between a model answer and a student answer.

Performance

The proposed LSTM-EMD-SVOR system set new state-of-the-art marks:

  • Accuracy: On the Mohler dataset, it achieved a Mean Absolute Error (MAE) of 0.490, a massive improvement over the original baseline (0.978) and contemporary neural models like ABCNN (0.74).
  • Correlation: It achieved a Pearson’s $r$ of 0.649 on the Mohler dataset.
  • Generalization: On the SemEval "Unseen Questions" and "Unseen Domains" tests, the model showed strong transferability, maintaining lower error rates than simpler Siamese architectures.

Conclusion

By treating grading as a concept-matching transportation problem, the LSTM-EMD-SVOR architecture offers a robust and interpretable solution for ASAG. While the Sinkhorn approximation manages computational complexity for short answers, future work may explore full backpropagation through the scaling steps and more nuanced label mapping for categorical datasets.

Summary updated

Abstract

Automatic short answer grading (ASAG) can reduce tedium for instructors, but is complicated by free-form student inputs. An important ASAG task is to assign ordinal scores to student answers, given some “model” or ideal answers. Here we introduce a novel framework for ASAG by cascading three neural building blocks: Siamese bidirectional LSTMs applied to a model and a student answer, a novel pooling layer based on earth-mover distance (EMD) across all hidden states from both LSTMs, and a flexible final regression layer to output scores. On standard ASAG data sets, our system shows substantial reduction in grade estimation error compared to competitive baselines. We demonstrate that EMD pooling results in substantial accuracy gains, and that a support vector ordinal regression (SVOR) output layer helps outperform softmax. Our system also outperforms recent attention mechanisms on LSTM states.
BibTeX
@inproceedings{kumar2017earth,
  author = {Sachin Kumar and Soumen Chakrabarti and Shourya Roy},
  title = {{Earth Mover's Distance Pooling over Siamese LSTMs for Automatic Short Answer Grading}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/284},
}
Figure