TL;DR
GHA-SmellFix is a new agentic LLM pipeline designed to automatically detect and refactor 19 types of "configuration smells" in GitHub Actions (GHA) workflows. The research reveals that while splitting detection and refactoring into separate agents doesn't always improve patch quality, it acts as a critical "gatekeeper" that helps models abstain from making unnecessary or dangerous changes to CI/CD pipelines.
What Problem Does This Paper Address?
Modern software relies heavily on GitHub Actions for CI/CD, but these workflows are often riddled with "configuration smells"—anti-patterns like unpinned action versions, missing timeouts, or unsafe artifact uploads that compromise security and reliability.
Existing solutions are mostly rule-based, making them brittle and difficult to maintain as GHA features evolve. Furthermore, most tools only report smells; they don't fix them. While Large Language Models (LLMs) are great at code transformation, using them for CI/CD is risky. A "hallucinated" edit in a workflow file could accidentally grant excessive permissions or break a deployment. The authors investigate whether an agentic approach—decomposing the task into specialized roles—can make LLM-driven refactoring safer and more effective.
Methods
The authors developed GHA-SmellFix, a pipeline implemented using the Agno framework. They compared two primary orchestration designs:
- Joint Design (D1): A single LLM agent receives the workflow and smell definition, then decides if a smell exists and provides a patch in one go.
- Decomposed Design (D2): A "Detection Agent" first classifies the smell. Only if it confirms the smell is present is a second "Refactoring Agent" invoked to generate a patch.
The pipeline was tested across four models: GPT-5.4-nano, Gemini 3.5 Flash, Claude Haiku 4.5, and DeepSeek V4 Flash. To evaluate the results, they used a "construct-aware" protocol that doesn't just check if the code changed, but specifically whether the LLM modified the correct YAML constructs (e.g., jobs.<id>.timeout-minutes) without touching unrelated, behavior-critical parts of the workflow.
Datasets
The study introduces a substantial new benchmark for GHA maintenance:
- Ground Truth Dataset: 1,710 manually curated (workflow, smell) pairs, covering 19 distinct smell types.
- Smell Catalog: The authors operationalized 19 smells from existing literature, ranging from "Localized" smells (visible in a single line) to "Context-Dependent" smells (requiring reasoning across the whole file).
- Evaluation Set: A balanced subset of 570 pairs (15 positive and 15 negative cases per smell) used for cross-model comparison.
Key Insights and Ideas
- Detection is the Bottleneck: The study found that once an LLM correctly identifies a smell, it is remarkably good at fixing it (92.6% clean removal rate). The primary failure mode isn't bad refactoring, but rather failing to see the smell or "hallucinating" one where none exists.
- Decomposition as a Gatekeeper: Splitting the agents (D2) didn't necessarily make the patches "smarter," but it significantly improved abstention. By forcing a dedicated detection step, the system was less likely to make "unnecessary edits" on clean files.
- The "Clean-up" Impulse: LLMs have a tendency to over-refactor. Even when fixing a specific smell, they often deleted helpful comments or changed unrelated versions. In a CI/CD context, these "cosmetic" changes are actually high-risk.
- Efficiency Gains: Counter-intuitively, the decomposed design (D2) often reduced latency and API costs. By using a detection "gate," the system avoids calling the more expensive refactoring logic for the many workflows that are already smell-free.
Key Results
- Improved Accuracy: The decomposed design (D2) raised the average F1-score from 86.34% to 87.55% and recall from 79.12% to 82.37%.
- Refactoring Success: For correctly detected smells, the models achieved a 74% clean removal rate overall, which jumped to 92.6% when conditioned on correct detection.
- Abstention Challenges: Despite the benefits of decomposition, the pipelines still produced 93 unnecessary edits on negative cases. Over half of these (54.84%) affected behavior-relevant constructs, highlighting the need for human-in-the-loop review.
Limitations and Future Work
The authors acknowledge that their evaluation is construct-aware but not runtime-semantic. They verified that the YAML was structurally correct and the smell was removed, but they did not actually execute the workflows to ensure no subtle runtime regressions occurred.
Future work includes integrating external validators (like linters or sandboxed CI runs) into the agentic loop and exploring "multi-smell" refactoring, where an agent must coordinate fixes for several overlapping issues in a single complex workflow. For now, they recommend that GHA-SmellFix be used as a review-support tool rather than a fully autonomous "auto-fixer."
Summary updated