This paper introduces BeanCounter, a massive, high-quality dataset designed to help large language models (LLMs) escape the "toxicity trap" of the open web. By leveraging decades of official business disclosures, BeanCounter provides a factual, highly structured, and significantly less toxic alternative to traditional web-scraped data.
The Problem: The Double-Edged Sword of Web Data
The "scaling laws" of LLMs have created an insatiable hunger for training data. To date, this hunger has been fed primarily by scraping the open web (e.g., Common Crawl). However, web data is riddled with personally identifiable information (PII), hate speech, factual errors, and increasingly, low-quality machine-generated text.
The authors argue that the AI community desperately needs novel sources of large-scale, professional, and factual text. To solve this, they turned to the world of corporate finance—specifically the SEC’s EDGAR system—which contains decades of legally mandated disclosures. Unlike a Reddit thread or a random blog post, these documents are written by professionals and certified by executives under penalty of law, making them a goldmine for high-quality training data.
Methods: Building the BeanCounter Pipeline
The authors built a sophisticated data processing pipeline to transform raw SEC filings into a machine-learning-ready format.
- Collection: They downloaded the entire history of EDGAR submissions (1996–2023), totaling 3TB of compressed data.
- Extraction: SEC filings use a mix of legacy SGML and modern HTML. The authors developed custom extractors that preserve crucial structural information—like indentation and list formats—which are vital for understanding complex business logic but are often lost in previous financial datasets.
- Cleaning & Filtering: They applied a "light touch" cleaning process. Because the source is professional, they avoided the aggressive (and often destructive) filters used on web data, simply excluding documents with high whitespace ratios or very low word counts.
- Deduplication: Using MinHash and Locality Sensitive Hashing (LSH), they removed near-duplicate documents to finalize the corpus.
- Toxicity Analysis: They used the Perspective API to compare BeanCounter against the C4 dataset, specifically analyzing how demographic identities are discussed in corporate versus web contexts.
The Datasets
The pipeline resulted in a massive, structured corpus ready for LLM pre-training and fine-tuning.
- BeanCounter: The primary contribution, containing 159.4 billion cleaned tokens (110.5 billion after deduplication).
- BeanCounter.fraud: A unique 308-million-token subset containing filings from companies later caught in SEC enforcement actions (e.g., Enron, Wirecard). This split is intended for research into detecting deceptive text.
Key Insights and Results
- The "Boring" Advantage (Toxicity Reduction): The most striking finding is the toxicity gap. Sentences in BeanCounter containing demographic descriptors are 59–89% less toxic than those in C4. In corporate filings, mentions of demographic groups usually occur in the context of non-discrimination policies or workforce statistics, rather than the vitriol found on the open web. Continually pre-training Pythia-1.4B and Phi-1.5 on just 1 billion tokens of BeanCounter led to an 18–33% reduction in toxic generation.
- Dark Matter Data: Less than 0.1% of BeanCounter appears in C4. This means BeanCounter provides a massive amount of "novel" information that current LLMs haven't seen during standard pre-training.
- Domain Expertise: The Phi-1.5 model saw a 4.3% improvement on Financial Named Entity Recognition (Fin NER) and a 2.5% boost in financial sentiment analysis after training on the dataset.
- Temporal Grounding: Unlike web pages, every document in BeanCounter has a precise, second-accurate timestamp. This allows researchers to study how models handle time-dependent facts and knowledge cutoffs.
- General Reasoning Maintained: Interestingly, the models maintained most of their general reasoning capabilities (MMLU, BBH), suggesting that domain-specific pre-training doesn't necessarily lead to catastrophic forgetting of general knowledge.
Limitations and Future Work
While highly promising, the authors note a few limitations. First, the evaluation was limited to relatively small models (up to 1.4B parameters); the impact on 70B+ parameter models remains to be tested. Second, while corporate filings are legally "factual," they are also heavily curated by PR departments. The BeanCounter.fraud split is a starting point, but more work is needed to help models distinguish between professional reporting and sophisticated deception. Finally, while low in toxicity, the dataset is inherently skewed toward Western identities and English-speaking markets, reflecting the SEC's jurisdiction.
Summary updated