SECOMlint: A linter for Security Commit Messages

Sofia Reis, Corina S. Pasareanu, Rui Abreu, Hakan ErdogmusarXiv 2023 · OpenReview

TL;DR

This paper introduces SECOMlint, an automated tool that checks security commit messages against the SECOM standard, aiming to improve the clarity and completeness of vulnerability disclosure. By leveraging Natural Language Processing (NLP) techniques like Named-Entity Recognition (NER), SECOMlint helps developers write better commit messages, leading to more transparent and efficient vulnerability management.

What Problem Does This Paper Address?

The security community faces a significant challenge in transparently disclosing vulnerabilities and their corresponding patches. This is largely due to poor documentation quality, stemming from a lack of standardized practices. Many security commit messages are brief and lack crucial details about the vulnerability, its impact, and the fix. This deficiency hinders maintenance teams, researchers, and even automated tools from fully understanding and managing security issues. The SECOM convention was proposed to address this by defining a structured format for security commit messages, but enforcing this standard manually is time-consuming and error-prone.

Methods

SECOMlint acts as an automated linter to enforce the SECOM convention. Its core functionality relies on Named-Entity Recognition (NER), a natural language processing technique. The tool parses commit messages, breaking them down into predefined sections like header, body, metadata, contacts, and bug-tracker references. For each section, SECOMlint employs a custom NER extractor, built using the spaCy library, to identify and label specific security-related entities (e.g., vulnerability IDs, weakness types, severity levels, reporter names). These extracted entities are then matched against a set of rules defined by the SECOM convention. The tool generates a compliance report, highlighting violations and providing a compliance score. SECOMlint is also configurable, allowing users to adjust the importance of rules and customize checks.

Datasets

The paper mentions evaluating SECOMlint on two datasets:

  1. pre_secom: A subset of 500 random security commit messages collected from the NVD and OSV databases, excluding automated entries.
  2. after_secom: A subset of 500 random security commit messages from an experiment where a security engineer documented patches using the SECOM convention.

These datasets were used to assess the tool's effectiveness in identifying SECOM compliance and the richness of information extractable from commit messages under different documentation practices.

Key Insights and Ideas

  • The critical role of commit message quality: The paper strongly emphasizes that well-structured commit messages are not just a matter of good practice but are essential for effective vulnerability management, research, and tool integration.
  • NLP as a powerful enforcer: SECOMlint demonstrates how NLP techniques, specifically NER, can be effectively applied to automate the enforcement of structured documentation standards like SECOM.
  • SECOM convention improves information extraction: The evaluation shows that when the SECOM convention is followed, SECOMlint can extract significantly more entities per commit message, indicating richer and more structured information is available.
  • Customization is key: The ability to configure rules and their importance makes SECOMlint adaptable to different team needs and workflows.

Key Results

  • SECOMlint was evaluated on two distributions of commit messages.
  • The pre_secom sample (messages not following SECOM) had a mean compliance score of 76.01%. The tool extracted an average of 5 entities per message.
  • The after_secom sample (messages following SECOM) had a mean compliance score of 86.61%. The tool extracted an average of 20 entities per message.
  • This indicates a substantial improvement in both compliance and the amount of extractable security-relevant information when the SECOM convention is applied.

Limitations and Future Work

Limitations: * The paper doesn't explicitly detail limitations of the NER model itself, such as potential misclassifications or missed entities, which are inherent to NLP tasks. * While the tool extracts information, it doesn't automatically fix the commit messages, requiring human intervention.

Future Work: The authors plan to:

  • Make the tool more customizable and extensible.
  • Develop features to generate suggestions for writers.
  • Explore potential for automated refactoring of commit messages.

Summary updated

Abstract

Transparent and efficient vulnerability and patch disclosure are still a challenge in the security community, essentially because of the poor-quality documentation stemming from the lack of standards. SECOM is a recently-proposed standard convention for security commit messages that enables the writing of well-structured and complete commit messages for security patches. The convention prescribes different bits of security-related information essential for a better understanding of vulnerabilities by humans and tools. SECOMlint is an automated and configurable solution to help security and maintenance teams infer compliance against the SECOM standard when submitting patches to security vulnerabilities in their source version control systems. The tool leverages the natural language processing technique Named-Entity Recognition (NER) to extract security-related information from commit messages and uses it to match the compliance standards designed. We demonstrate SECOMlint at https://youtu.be/-1hzpMN_uFI; and documentation and its source code at https://tqrg.github.io/secomlint/.
BibTeX
@misc{reis2023secomlint,
  author = {Sofia Reis and Corina S. Pasareanu and Rui Abreu and Hakan Erdogmus},
  title = {{SECOMlint: A linter for Security Commit Messages}},
  year = {2023},
  eprint = {2301.06959},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.06959},
}
Figure