TL;DR
This paper explores the potential of Large Language Models (LLMs) to identify and analyze tax-minimization strategies, a complex domain typically requiring expert human knowledge. The authors introduce a novel dataset and benchmark LLMs on their ability to interpret, complete, and generate tax strategies, finding that while LLMs are generally mediocre, they show flashes of brilliance, with one LLM even generating a novel tax strategy.
What Problem Does This Paper Address?
The U.S. government loses an estimated $600 billion annually due to tax-minimization schemes, often employed by well-advised, wealthy taxpayers. Identifying these schemes is challenging, even for human experts, due to the sheer volume and complexity of U.S. tax law, which spans hundreds of thousands of pages of statutes, case law, and administrative guidance. This complexity makes it an ideal domain for AI, provided AI can reason over legal texts, understand strategy mechanics, and generate novel strategies. The paper aims to evaluate the capabilities of current advanced LLMs in this domain, bridging the gap between AI's potential and the practical challenges of tax law enforcement.
Methods
The researchers developed a novel dataset called "Shelter Check," comprising 36 known U.S. tax-minimization strategies derived from public sources. Each strategy includes the relevant tax authorities, background facts, goals, step-by-step implementation, and a legal analysis. To test LLMs, they designed three main tasks:
- Analysis Verification: LLMs were prompted to determine if the legal analysis provided for each strategy step was "viable" (having a reasonable basis) or "correct" (legally sound).
- Goal Verification: LLMs assessed whether the strategies met their stated goals, both with and without the provided legal analysis. An "adversarial step" was introduced to deliberately break strategies, testing the LLMs' ability to identify non-viable schemes.
- Strategy Generation:
- Step-Cloze Task: LLMs were asked to fill in a single missing step in a partially defined strategy.
- From-Scratch Generation: LLMs were prompted with the background, goals, and authorities of existing strategies to generate entirely new strategies.
The LLMs were evaluated on their accuracy in these tasks, with a focus on how well their outputs aligned with domain expert assessments.
Datasets
The primary dataset introduced is Shelter Check, containing 36 U.S. tax-minimization strategies. These strategies are derived from public sources, including IRS "listed transactions," high-profile court decisions, and legislative flaws. The dataset is structured to include:
- Relevant tax-law authorities (U.S. tax code sections, Treasury Regulations, revenue rulings, tax treaties, case law).
- Background facts necessary for a taxpayer to use the strategy.
- The goals of the strategy (e.g., minimizing taxes, receiving cash).
- Concrete, step-by-step instructions for implementing the strategy.
- A legal analysis explaining why the strategy meets its goals.
- An adversarial step designed to make the strategy fail.
This dataset is novel in its focus on real-world tax avoidance strategies and its detailed breakdown for LLM evaluation, differing from existing legal NLP datasets that often focus on classification or retrieval tasks.
Key Insights and Ideas
- LLMs struggle with nuanced legal reasoning: While LLMs can interpret and verify legal analysis to some extent, they often fall short of human expert judgment, especially when assessing correctness rather than mere viability.
- The "viable" standard is more accessible to LLMs: LLMs performed better when asked if a strategy had a "reasonable basis" (viable) compared to whether it was legally "correct." This highlights a potential gap in their ability to perform rigorous legal validation.
- LLMs can be overly optimistic (agreement bias): Notably, models like Claude-4 showed a tendency to find strategies viable even when an adversarial step was introduced to make them fail, suggesting an "agreement bias" that could limit their usefulness in identifying non-viable schemes.
- Novel strategy generation is a breakthrough: The most significant finding is that one LLM (o1-preview) generated a completely novel tax strategy that was deemed legally sound and met its goals by a domain expert. This demonstrates LLMs' potential to move beyond pattern recognition to genuine creative problem-solving in complex domains.
- Memorization is limited: Despite being trained on vast amounts of data, LLMs did not simply regurgitate known strategies, indicating that their ability to generate novel solutions is not solely based on memorization.
Key Results
- Analysis Verification: LLMs showed high accuracy in verifying analysis steps as "viable" (e.g., o3 and Claude-4 achieved 193/193 and 170/193 correct analyses, respectively). However, accuracy dropped for the "correct" standard. Gemini-2.5 performed worse, indicating issues with its understanding of legal nuances.
- Goal Verification: LLMs were generally good at verifying goals when provided with human analysis. Claude-4 and o3 verified all 36 strategies as viable with analysis. Without analysis, performance dipped for most models.
- Adversarial Step Goal-Failure Verification: LLMs struggled significantly to identify strategies that failed due to the adversarial step. Models often found these broken strategies viable or even correct, with Claude-4 showing a strong agreement bias (20/36 viable). The baseline Llama-3.3-70B model performed worse overall but also showed agreement bias.
- Step-Cloze Task: LLMs achieved mediocre performance, with grades around 2 out of 3 on average for filling in missing strategy steps. Performance improved with few-shot prompting (1-shot) but slightly decreased with more examples (2-shot).
- From-Scratch Strategy Generation: LLMs generated strategies with a mean grade of 1.47 (o1-preview) and 1.23 (claude-3.5) on a 0-3 scale. O1-preview generated more strategies with the highest grade (3/11 vs. 7/36 for Claude-3.5), including the novel strategy.
- Model Grading: LLMs showed poor agreement with domain expert grades when evaluating entire generated strategies, with Spearman's ρ values around 0.30-0.48, suggesting strategy-level grading is much harder than step-level grading.
Limitations and Future Work
- Dataset Size: The Shelter Check dataset is small (36 strategies) due to the significant domain expert time required for its creation.
- Jurisdictional Scope: The dataset and findings are U.S.-centric and may not generalize to other legal systems.
- Subjectivity in Grading: Legal judgment calls are inherently debatable, introducing potential subjectivity into dataset creation and evaluation.
- Limited Model Testing: Only four LLMs were tested.
Future work could involve expanding the dataset, testing more models, and exploring how LLMs can be further refined to overcome agreement bias and improve their ability to rigorously validate legal strategies. The authors also highlight the ethical considerations of this research, aiming to aid tax agencies but acknowledging the potential for misuse by taxpayers.
Summary updated