Create Your Own Work Stack: Deploying Self-hosted AI and Tooling in a Tech Startup

Saad Ahmed Bazaz, Talha Amjad, Abdurrehman Subhani, Sohaib Ahmed Bazaz, Abdullah AamirarXiv 2026 · arXiv

TL;DR

This paper details a real-world case study of a small tech startup transitioning from a purely commercial AI workflow to a hybrid self-hosted stack. By offloading "coordination tasks"—like syncing documentation and managing tickets—to local models while reserving commercial agents for complex coding, the team achieved a 27–36% reduction in commercial AI costs without sacrificing quality.

What Problem Does This Paper Address?

As AI-assisted development becomes the norm, small teams are facing a "success tax": the more they use powerful commercial models like Claude or Codex, the higher their monthly bills climb. At Grayhat Developers, commercial AI usage reached over 25 billion tokens per month, costing roughly $325—a significant recurring expense for a team of under 40 people.

The authors set out to see if self-hosted, open-source models could replace these expensive commercial services. However, they quickly discovered that local models often lack the "reasoning horsepower" for complex repository-level coding. The challenge shifted from replacement to orchestration: how can a small team build a hybrid infrastructure that uses the right model for the right task while minimizing "UX friction" that often kills internal tool adoption?

Methods

The researchers implemented a hybrid architecture they call a "Work Stack." Key components included:

  • The Infrastructure: They used two separate machines—one for self-hosted tools (Outline for docs, Plane for task management, ERPNext for ERP) and a dedicated AI Workload Machine equipped with an NVIDIA RTX 4080.
  • The Models: They initially experimented with models like Kimi and DeepSeek but eventually settled on Gemma 3 (starting with the 2.7B parameter version) deployed via llama.cpp and managed by GPUStack.
  • The Protocol: They utilized the Model Context Protocol (MCP) to connect their AI systems to their internal tools, allowing the models to retrieve and summarize organizational context.
  • The Interface Iteration: They moved from a decentralized setup (LobeHub) where users had to configure their own connections, to a centrally managed LibreChat deployment. This "version 2" pre-configured all tool connectors, significantly lowering the barrier to entry for the team.
Figure 1: Overview of the hybrid AI workflow process. The self-hosted AI resembles a "moat" around the or- ganization’s internal tools.

Datasets

The paper does not involve training on public datasets. Instead, it uses internal operational data from Grayhat Developers:

  • Token Usage Logs: Recorded activity exceeding 25.93 billion tokens.
  • Qualitative Feedback: Structured and unstructured interviews with 11 employees to categorize issues with the AI stack (e.g., setup, performance, reliability).

Key Insights and Ideas

  • The "Coordination Workload" is the Low-Hanging Fruit: The authors found that 30–40% of their AI usage wasn't actually coding; it was "glue work"—summarizing docs, moving context between tools, and creating tickets. Local models are perfectly capable of handling ~90% of this work.
  • UX is the Ultimate Gatekeeper: Even a team of engineers will abandon a tool if the setup is too complex. The transition from v1 to v2 showed that centralizing configuration was more important for adoption than the raw performance of the underlying model.
  • The "Moat" Strategy: By keeping organizational context (docs, tasks, ERP data) within a self-hosted AI layer, the company improved data privacy. Only the specific, high-complexity coding tasks were sent to external commercial providers.
  • The Kübler-Ross Analogy: The authors humorously but accurately map their journey to the stages of grief: Denial (thinking AI would just be cheap), Anger (frustration with early open-source models), Bargaining (settling on a hybrid approach), and finally Acceptance.

Key Results

The hybrid approach yielded significant quantitative and qualitative improvements:

  • Cost Displacement: The team estimated a 27–36% reduction in overall commercial AI workload, translating to savings of $88–$117 per month.
  • Operating Costs: The local LLM cost only $10–$30/month in electricity and minor maintenance, far below the cost of the commercial tokens it replaced.
  • User Adoption: Version 2 (LibreChat) saw a 62.5% decrease in complaints regarding "Setup & Onboarding" and a 150% increase in positive/neutral sentiment compared to the initial deployment.
Table 4: Estimated commercial-AI cost-equivalent ex- posure avoided after introducing a local LLM.

Limitations and Future Work

  • Manual Routing: Currently, humans must decide which model to use for a specific task. The authors want to implement an automated routing system (using a lightweight classifier) to direct queries to the most cost-effective model automatically.
  • Infrastructure Overhead: While token costs went down, engineering "observability" costs went up. The team had to spend time setting up monitoring tools like Beszel to ensure server uptime.
  • Integration Gaps: The local agents and commercial coding agents are still somewhat siloed. Future work aims to let these agents communicate directly, allowing a commercial agent to "ask" a local agent for organizational context without human intervention.
  • WorkStack Deployment: The authors plan to release a one-click deployment version of this environment to help other startups replicate their results.

Summary updated

Abstract

Self-hosting AI is often presented as a way to reduce inference costs and increase control, but deployment introduces its own costs. We report our experience deploying a hybrid AI stack at Grayhat Developers, a small software company with an AI-intensive development workflow. Our initial motivation was to counter the growing cost of commercial AI usage: Claude/Codex incurred approximately \$325 per month, with more than 25.93 billion recorded tokens. We experimented with local open-source models as replacements for commercial coding agents, but after learning that their performance and inference speed were insufficient, we adopted a hybrid approach in which commercial agents remained responsible for complex tasks while local models handled lower-complexity tasks, organizational information retrieval, and synchronization with self-hosted tools including Outline, Plane, and ERPNext. We estimated that approximately 30--40\% of our commercial AI workload was associated with context transfer, documentation synchronization, task management, and related operational coordination. By moving this work to local models, we estimate that approximately 90\% of this coordination workload could be handled without commercial inference, corresponding to an estimated 27--36\% reduction in overall commercial-AI workload or cost-equivalent usage. Our first internal deployment nevertheless failed to achieve adoption because it introduced UX friction. Replacing it with a centrally configured deployment substantially improved feedback and adoption. We conclude that local inference is most useful not as a wholesale replacement for commercial AI, but as part of a hybrid architecture that assigns tasks according to their capability requirements, cost, context demands, and operational sensitivity.
BibTeX
@misc{bazaz2026create,
  author = {Saad Ahmed Bazaz and Talha Amjad and Abdurrehman Subhani and Sohaib Ahmed Bazaz and Abdullah Aamir},
  title = {{Create Your Own Work Stack: Deploying Self-hosted AI and Tooling in a Tech Startup}},
  year = {2026},
  eprint = {2611.08116},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08116},
}
Figure