BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text

Siyan Wang, Bradford LevyNeurIPS 2024 · OpenReview

This paper introduces BeanCounter, a massive, high-quality dataset designed to help large language models (LLMs) escape the "toxicity trap" of the open web. By leveraging decades of official business disclosures, BeanCounter provides a factual, highly structured, and significantly less toxic alternative to traditional web-scraped data.

The Problem: The Double-Edged Sword of Web Data

The "scaling laws" of LLMs have created an insatiable hunger for training data. To date, this hunger has been fed primarily by scraping the open web (e.g., Common Crawl). However, web data is riddled with personally identifiable information (PII), hate speech, factual errors, and increasingly, low-quality machine-generated text.

The authors argue that the AI community desperately needs novel sources of large-scale, professional, and factual text. To solve this, they turned to the world of corporate finance—specifically the SEC’s EDGAR system—which contains decades of legally mandated disclosures. Unlike a Reddit thread or a random blog post, these documents are written by professionals and certified by executives under penalty of law, making them a goldmine for high-quality training data.

Methods: Building the BeanCounter Pipeline

The authors built a sophisticated data processing pipeline to transform raw SEC filings into a machine-learning-ready format.

Data processing pipeline for BeanCounter
  1. Collection: They downloaded the entire history of EDGAR submissions (1996–2023), totaling 3TB of compressed data.
  2. Extraction: SEC filings use a mix of legacy SGML and modern HTML. The authors developed custom extractors that preserve crucial structural information—like indentation and list formats—which are vital for understanding complex business logic but are often lost in previous financial datasets.
  3. Cleaning & Filtering: They applied a "light touch" cleaning process. Because the source is professional, they avoided the aggressive (and often destructive) filters used on web data, simply excluding documents with high whitespace ratios or very low word counts.
  4. Deduplication: Using MinHash and Locality Sensitive Hashing (LSH), they removed near-duplicate documents to finalize the corpus.
  5. Toxicity Analysis: They used the Perspective API to compare BeanCounter against the C4 dataset, specifically analyzing how demographic identities are discussed in corporate versus web contexts.

The Datasets

The pipeline resulted in a massive, structured corpus ready for LLM pre-training and fine-tuning.

Distribution of tokens by year and form type
  • BeanCounter: The primary contribution, containing 159.4 billion cleaned tokens (110.5 billion after deduplication).
  • BeanCounter.fraud: A unique 308-million-token subset containing filings from companies later caught in SEC enforcement actions (e.g., Enron, Wirecard). This split is intended for research into detecting deceptive text.

Key Insights and Results

  • The "Boring" Advantage (Toxicity Reduction): The most striking finding is the toxicity gap. Sentences in BeanCounter containing demographic descriptors are 59–89% less toxic than those in C4. In corporate filings, mentions of demographic groups usually occur in the context of non-discrimination policies or workforce statistics, rather than the vitriol found on the open web. Continually pre-training Pythia-1.4B and Phi-1.5 on just 1 billion tokens of BeanCounter led to an 18–33% reduction in toxic generation.
  • Dark Matter Data: Less than 0.1% of BeanCounter appears in C4. This means BeanCounter provides a massive amount of "novel" information that current LLMs haven't seen during standard pre-training.
  • Domain Expertise: The Phi-1.5 model saw a 4.3% improvement on Financial Named Entity Recognition (Fin NER) and a 2.5% boost in financial sentiment analysis after training on the dataset.
  • Temporal Grounding: Unlike web pages, every document in BeanCounter has a precise, second-accurate timestamp. This allows researchers to study how models handle time-dependent facts and knowledge cutoffs.
  • General Reasoning Maintained: Interestingly, the models maintained most of their general reasoning capabilities (MMLU, BBH), suggesting that domain-specific pre-training doesn't necessarily lead to catastrophic forgetting of general knowledge.

Limitations and Future Work

While highly promising, the authors note a few limitations. First, the evaluation was limited to relatively small models (up to 1.4B parameters); the impact on 70B+ parameter models remains to be tested. Second, while corporate filings are legally "factual," they are also heavily curated by PR departments. The BeanCounter.fraud split is a starting point, but more work is needed to help models distinguish between professional reporting and sophisticated deception. Finally, while low in toxicity, the dataset is inherently skewed toward Western identities and English-speaking markets, reflecting the SEC's jurisdiction.

Summary updated

Abstract

Many of the recent breakthroughs in language modeling have resulted from scaling effectively the same model architecture to larger datasets. In this vein, recent work has highlighted performance gains from increasing training dataset size and quality, suggesting a need for novel sources of large-scale datasets. In this work, we introduce BeanCounter, a public dataset consisting of more than 159B tokens extracted from businesses' disclosures. We show that this data is indeed novel: less than 0.1% of BeanCounter appears in Common Crawl-based datasets and it is an order of magnitude larger than datasets relying on similar sources. Given the data's provenance, we hypothesize that BeanCounter is comparatively more factual and less toxic than web-based datasets. Exploring this hypothesis, we find that many demographic identities occur with similar prevalence in BeanCounter but with significantly less toxic context relative to other datasets. To demonstrate the utility of BeanCounter, we evaluate and compare two LLMs continually pre-trained on BeanCounter with their base models. We find an 18-33% reduction in toxic generation and improved performance within the finance domain for the continually pretrained models. Collectively, our work suggests that BeanCounter is a novel source of low-toxicity and high-quality domain-specific data with sufficient scale to train multi-billion parameter LLMs.
BibTeX
@inproceedings{wang2024beancounter,
  author = {Siyan Wang and Bradford Levy},
  title = {{BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=HV5JhUZGpP},
}
Figure