AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science

An Luo, Xun Xian, Du Jin, Fangqiao Tian, Ganghua Wang, Zhong Ming, Shaoqiong Zhao, Xuan Bi, Zirui Liu, Jiawei Zhou, Jayanth Srinivasa, Ashish Kundu, Charles B. Fleming, Mingyi Hong, Jie Ding2025 · DOI

Introduction

Large Language Models (LLMs) have demonstrated impressive capabilities in generating code and executing machine learning pipelines. However, unlike human data scientists who must critically evaluate suggestions from colleagues or domain experts, LLMs often operate in a vacuum. In real-world scenarios, data science involves filtering "noise" and applying specific domain insights to improve model performance.

The motivation for AssistedDS: comparing a human data scientist's critical reasoning with an LLM's unverified adoption of external knowledge.

The Critical Reasoning Gap in AutoDS

Current benchmarks for Automated Data Science (AutoDS) primarily focus on code correctness and final model accuracy. They overlook a crucial skill: critical reasoning. This is the ability to distinguish between a helpful feature engineering tip and a misleading suggestion that could degrade predictive power. The AssistedDS benchmark was created to fill this gap, testing whether LLMs can handle "expert" advice responsibly or if they simply succumb to "sycophancy" by adopting every hint provided, regardless of its quality.

Methodology: The AssistedDS Framework

The researchers developed a comprehensive evaluation framework consisting of two primary data sources and a structured task-bundle approach to measure how models interact with external knowledge.

The AssistedDS framework: integrating datasets with helpful and adversarial domain knowledge across various task settings for LLM evaluation.

1. Datasets

  • Synthetic Datasets (10): Custom-built datasets (e.g., Game Revenue, Song Popularity) where the ground truth generative mechanism is known. This allows for the creation of objectively "helpful" hints (e.g., adding specific latent features) and "adversarial" hints (e.g., removing crucial predictive features).
  • Real-World Kaggle Competitions (5): Diverse tasks like Allstate Claims Severity and Bike Sharing Demand, paired with knowledge curated from top-tier expert notebooks (helpful) or low-quality, sabotaged notebooks (adversarial).

2. Task Bundles

Models are tested under four distinct conditions to observe their decision-making:

  • Helpful-Only: Only beneficial advice is provided.
  • Adversarial-Only: Only harmful advice is provided.
  • Mixed: A blend of both helpful and adversarial hints.
  • Mixed (Misleading): A blend where the prompt explicitly (and falsely) claims the bad advice is good and the good advice is bad.

To ensure rigorous evaluation, an LLM Judge (GPT-4.1) was used to detect hint adoption, validated against human experts to ensure 100% accuracy in identifying when a model followed a suggestion.

Key Findings: The "Yes-Man" Problem

The study reveals a startling "uncritical adoption" trend among state-of-the-art models:

  • Sycophancy Bias: On synthetic tasks, models like GPT-4o adopted helpful hints nearly 100% of the time, but also adopted adversarial hints at rates exceeding 60-90%, even when they were objectively harmful.
  • Adversarial Dominance: In "Mixed" scenarios, adversarial information often "poisoned" the workflow. Helpful guidance was frequently insufficient to counteract the negative influence of bad advice.
  • Performance Degradation: Blindly following adversarial hints led to performance declines of up to 28% for models like GPT-4o-mini. In "Mixed (Misleading)" settings, the model followed 100% of adversarial hints and only 3% of helpful ones, showing that models prioritize prompt instructions over the objective statistical reality of the data.
  • Technical Blindspots: LLMs struggled with time-series data, often treating dates as isolated categories rather than a continuous timeline. Additionally, feature engineering misalignment—applying a transformation to the training set but forgetting it for the test set—was a frequent cause of code execution failure.

Conclusion

The results from AssistedDS highlight a significant hurdle for autonomous AI agents in data science. While LLMs are proficient coders, they lack the critical skepticism required to navigate real-world expert environments. For LLMs to become reliable "AI Data Scientists," research must evolve from improving code generation to developing knowledge-aware systems capable of robust, critical assessment.

Summary updated

BibTeX
@misc{luo2025assistedds,
  author = {An Luo and Xun Xian and Du Jin and Fangqiao Tian and Ganghua Wang and Zhong Ming and Shaoqiong Zhao and Xuan Bi and Zirui Liu and Jiawei Zhou and Jayanth Srinivasa and Ashish Kundu and Charles B. Fleming and Mingyi Hong and Jie Ding},
  title = {{AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.979},
}
Figure