TL;DR
AI coding assistants are widely expected to democratize software development, but this paper reveals a hidden structural bias: they perform significantly better on mainstream languages and frameworks than on niche ones. By evaluating five state-of-the-art LLMs across thousands of tasks, the authors demonstrate a "Matthew Effect" ("the rich get richer") in software evolution, where data-rich ecosystems receive superior AI support while niche technologies face a steep "AI productivity tax."
What Problem Does This Paper Address?
While AI programming tools (like GitHub Copilot, Cursor, and various LLMs) have popularized new paradigms like "vibe coding" and "agentic coding," their impact on the broader software ecosystem is highly uneven. There has been an optimistic assumption that LLMs act as a "Great Equalizer," making specific syntax irrelevant. However, LLMs are trained on public code repositories, which are overwhelmingly dominated by popular languages and frameworks.
This paper investigates whether this data imbalance creates a self-reinforcing feedback loop—a Matthew Effect—where popular technologies become even more dominant because AI tools support them better. This effectively penalizes developers who choose niche or emerging technologies that might be technically superior for a given task, potentially stifling long-term innovation and diversity in software engineering.
Methods
The authors designed a two-tier experimental pipeline to evaluate both language-level algorithmic proficiency and framework-level application development:
- Algorithmic Task Evaluation: They automated the generation and testing of code across 9 different programming languages using 5 state-of-the-art LLMs. They built a multi-stage code extraction and cleaning pipeline to isolate pure executable code from LLM responses and evaluated them using a distributed submission system.
- Framework Task Evaluation: They simulated real-world "vibe coding" and "agentic coding" protocols using commercial tools (Cursor, CodeBuddy, Copilot). The AI acted as an autonomous agent tasked with building full-stack applications, with human intervention strictly limited to feeding back raw error messages in an iterative debugging loop until the code worked or failed.
The overall two-tier pipeline architecture is illustrated below:
Datasets
- LeetCode Benchmark: A dataset of 3,011 public, non-paid problems from 2025 (765 easy, 1,526 medium, 720 hard) evaluated across 9 languages, totaling 135,495 code generation requests. Using recent 2025 problems helped minimize "rote recall" from training data contamination.
- Full-Stack Framework Benchmark: 17 generic CRUD application scenarios (e.g., movie ticket booking, library management) implemented across 6 full-stack combinations ranging from mainstream (Vue + Spring Boot) to emerging/niche (SolidJS + Actix Web).
- Divergent Technology Pathways: 8 specialized architectural scenarios (e.g., high-concurrency API gateways, real-time chat servers) explicitly designed to pit mainstream, middle-ground, and niche stacks against each other.
Key Insights and Ideas
- The Popularity Premium: Language popularity is a stronger predictor of AI coding success than the underlying capability of the model itself. Even the most advanced models struggle with basic syntax in lower-resource languages.
- The Complexity Penalty: The performance gap between mainstream and niche languages scales non-linearly with task difficulty. On easy tasks, the gap is noticeable, but on hard tasks, niche languages drop almost to a 0% success rate while mainstream languages maintain solid performance.
- Syntax vs. Logic Failures: When mainstream languages fail, it's usually due to logical errors (Wrong Answer or Runtime Error). When niche languages fail, they overwhelmingly fail at the compilation stage (Compile Error), meaning the LLM cannot even generate syntactically correct code.
- A Self-Reinforcing Loop: This creates a dangerous ecosystem dynamic. Because AI tools make mainstream stacks vastly more productive, developers are nudged away from niche alternatives, further starving those niche ecosystems of the public code data needed to train future AI models.
Key Results
- On algorithmic tasks, mainstream languages (Python, JavaScript, Java, C, C++) consistently achieved Pass@1 rates above 60% in top-performing models. In contrast, less popular languages like Erlang and Racket struggled dramatically, with success rates often below 25%. For example, DeepSeek-V3 achieved 79.81% for Python but only 24.31% for Erlang and 20.82% for Racket.
- The performance gap widens substantially as complexity increases. On Hard problems, top models achieved 50% to 63% success with popular languages, but plummeted to a mere 0% to 6% for less popular ones, as shown in the difficulty breakdown:
- In full-stack development, mainstream frameworks (Vue+Spring, React+Express, Django) typically converged successfully within 1–3 attempts. Niche frameworks (Svelte+FastAPI, SolidJS+Actix) required significantly more human intervention, often demanding 5–10 rounds of error-feeding or failing entirely.
Limitations and Future Work
- The study focuses heavily on LeetCode for algorithmic tasks, which does not fully capture the nuances of large-scale software engineering or repository-level context.
- The framework evaluation, while rigorous, relies on a fixed protocol of feeding back raw error messages, which may not perfectly mirror how an experienced human developer interacts with an AI assistant.
- Future Directions: The authors plan to expand the benchmark to broader domains, investigate multi-agent collaborative development dynamics, and explore "diversity-aware" training and inference strategies to counteract ecosystem homogenization.
Summary updated