TL;DR
This paper tackles the challenge of automatically classifying the topics of clauses within standard form contracts (like online shop T&Cs). They developed a novel taxonomy and a large, annotated dataset, then compared various NLP approaches, finding that BERT achieved the highest performance, though simpler models like logistic regression offered competitive results with significantly less computational cost.
What Problem Does This Paper Address?
Standard form contracts, ubiquitous in online commerce, are often lengthy and complex, making it difficult for consumers to understand their rights and obligations. While lawmakers have introduced regulations to protect consumers, the sheer volume of these contracts means most people don't read them. This paper addresses the need for automated tools to help users navigate these documents by classifying the topic of each clause. This classification can aid consumers in finding relevant information quickly and assist legal professionals in their work. The authors focus on Terms and Conditions (T&C) from online shops due to their economic relevance, public availability, and machine-readable format.
Methods
The authors employed a supervised learning approach, which necessitated the creation of a detailed taxonomy of clause topics. This taxonomy was developed collaboratively with legal experts, drawing from contract templates and industry standards. They then built a large, bilingual (German and English) corpus of over 6,000 clauses, manually annotated according to this taxonomy.
To classify these clauses, they compared seven different NLP approaches:
- Rule-based keyword matching: A baseline approach using lawyer-provided keywords.
- Logistic Regression: A classic machine learning model.
- Random Forest: An ensemble learning method.
- Multilayer Perceptron (MLP): A type of feedforward neural network.
- Long Short-Term Memory (LSTM): A recurrent neural network architecture.
- BERT (Bidirectional Encoder Representations from Transformers): A state-of-the-art transformer-based language model.
They trained separate models for topic and subtopic classification and experimented with various word embeddings (Word2Vec, GloVe) for the neural network approaches. Hyperparameter tuning was performed using k-fold cross-validation.
Datasets
The primary dataset used in this paper consists of over 6,000 clauses extracted from approximately 170 German and English online shop T&Cs. This dataset was manually annotated with topics and subtopics based on a newly developed taxonomy. The authors note that this corpus is protected by copyright and therefore cannot be published. They also mention using a larger, unannotated corpus of over 5,400 contracts for training domain-specific word embeddings.
Key Insights and Ideas
- BERT leads, but simpler models are competitive: While BERT achieved the highest F1-scores, simpler models like logistic regression performed surprisingly well, especially on the German corpus. This suggests that for practical applications where computational resources are a concern, simpler models might be a more suitable choice.
- Domain specificity matters: The taxonomy developed for online shop T&Cs showed limited coverage (33%) when applied to banking contracts, highlighting the need for domain-specific adaptations or broader taxonomies for different contract types.
- Keyword matching as a strong baseline: The rule-based keyword matching approach provided a solid baseline, particularly in German, where domain-specific compound nouns seem to aid classification.
- Multilingual approaches not beneficial (in this study): Contrary to some prior work, multilingual approaches did not improve performance in this specific task, warranting further investigation.
Key Results
The paper reports the following key results:
- BERT achieved the highest F1-scores: Up to 0.91 for topic classification in German.
- Logistic Regression performed strongly: Achieving F1-scores up to 0.87 for topics in German.
- German corpus yielded better results: Generally, all models performed better on the German corpus compared to the English one, which the authors attribute to the German corpus having longer clauses and more domain-specific compound nouns.
- Subtopic classification was more challenging: F1-scores for subtopic classification were consistently lower than for topic classification across all approaches.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Domain Specificity: The developed taxonomy and models are specific to online shop T&Cs. Future work could involve adapting these to other domains like banking, insurance, or employment contracts.
- Multilingual Models: The lack of improvement from multilingual models in this study warrants further investigation into why this might be the case, especially given prior research.
- Computational Cost: While BERT is the top performer, its computational demands for training and inference are significant. Exploring ways to make these models more efficient or leveraging simpler, yet effective, models like logistic regression for practical applications is a key consideration.
- Copyright Restrictions: The inability to publish the annotated corpus limits reproducibility and further research by the community.
Summary updated