Abstract
We present the system description for our submission towards the Key Point Analysis Shared Task at ArgMining 2021. Track 1 of the shared task requires participants to develop methods to predict the match score between each pair of arguments and keypoints, provided they belong to the same topic under the same stance. We leveraged existing state of the art pre-trained language models along with incorporating additional data and features extracted from the inputs (topics, key points, and arguments) to improve performance. We were able to achieve mAP strict and mAP relaxed score of 0.872 and 0.966 respectively in the evaluation phase, securing 5th place on the leaderboard. In the post evaluation phase, we achieved a mAP strict and mAP relaxed score of 0.921 and 0.982 respectively. All the codes to generate reproducible results on our models are available on Github.
Introduction
Key Point Analysis (KPA) is a vital task in quantitative summarization, aiming to condense large corpora of opinionated text into a manageable set of bullet points. In the context of the ArgMining 2021 shared task, the objective is to match crowd-sourced arguments to concise keypoints. This involves predicting a "match score" for argument-keypoint pairs within the same topic and stance. The core challenge is overcoming "semantic drift," where arguments are paraphrased in diverse ways, making simple lexical matching insufficient while requiring models to distinguish between related but distinct concepts.
Methodology
Team Enigma's approach treats keypoint matching as a sentence similarity task, utilizing large pre-trained Transformer models such as BERT, RoBERTa, BART, and DeBERTa.
The architecture, as shown above, incorporates several key innovations:
- Contextual Input Concatenation: To provide the model with maximum context, the input is formatted as a single string:
[Keypoint] + [Argument] + [Topic]. - Feature Injection: Beyond the Transformer's internal representations, the system extracts explicit features: Dependency Parse trees, Parts-of-Speech (POS) tags, and TF-IDF vectors. These features are encoded and concatenated with the Transformer's output embeddings.
- Neural Network Head: The concatenated embeddings are passed through two dense layers (256 and 128 neurons, respectively) followed by a sigmoid activation layer to produce the final similarity score.
- Intermediate Pre-training: Models were pre-trained on external datasets to enhance their semantic matching and argument evaluation capabilities before fine-tuning on the target task.
Data and Experimental Setup
The primary dataset for the task was ArgKP-2021, consisting of approximately 27,520 labeled argument/keypoint pairs across 31 controversial topics. To improve model robustness, two external datasets were used for intermediate pre-training:
- STS Benchmark: 8,020 sentence pairs for semantic textual similarity.
- IBM Rank 30k: 30,497 argument-topic pairs with quality scores, used to teach the model how to evaluate argument strength.
Results and Discussion
The system achieved a 5th-place finish on the leaderboard during the evaluation phase. Post-evaluation refinements, particularly the use of external data, significantly boosted performance.
Key Findings
- Impact of External Data: As shown in the results table, pre-training on the IBM Rank 30k dataset yielded the highest single-model performance, with BART-large reaching a strict mAP of 0.921.
- Lexical vs. Syntactic Features: TF-IDF features (lexical overlap) provided a more substantial performance boost than POS or Dependency Parsing. This suggests that while Transformers inherently learn syntax, explicit grounding in word overlap helps anchor semantic representations.
- Contextual Importance: Ablation studies highlighted that the "Topic" is an essential anchor; removing it from the input string led to a significant drop in performance.
- Model Performance: BART-large generally outperformed other models, likely due to its pre-training objectives that emphasize sentence-level context.
Conclusion
The Team Enigma system demonstrates the effectiveness of combining large pre-trained language models with explicit lexical features and intermediate pre-training on related tasks. While resource constraints limited the exploration of combined syntactic features, the results highlight the importance of context and lexical grounding in keypoint analysis. Future work will focus on incorporating natural language explanations to further guide the model's reasoning.
Summary updated