TL;DR
This paper introduces CAKE, the first benchmark specifically designed to evaluate Large Language Models' (LLMs) understanding of cloud-native software architecture. They found that while LLMs excel at multiple-choice questions about architecture, their ability to generate and explain architectural decisions (free-response) reveals significant gaps, especially for smaller models.
What Problem Does This Paper Address?
As LLMs increasingly act as "co-pilots" for software engineers, their ability to assist with complex architectural decisions is crucial. However, existing benchmarks primarily focus on code generation or general knowledge, leaving a gap in evaluating LLMs' grasp of software architecture principles, particularly in the rapidly evolving domain of cloud-native systems. This lack of evaluation makes it difficult for practitioners to know when and how reliably to trust LLM assistance for architectural tasks, which have significant implications for scalability, resilience, and maintainability.
Methods
The authors developed a novel benchmark called CAKE, which comprises 188 expert-validated questions covering five cloud-native topics (architectural patterns, quality attributes, decomposition strategies, cloud deployment, and technical debt) and four cognitive levels from Bloom's revised taxonomy: recall, analyze, design, and implement.
For evaluation, they tested 22 model configurations from four LLM families (Qwen, Llama, Mistral, GPT). Their evaluation pipeline combined two methods:
- Multiple-Choice Questions (MCQs): Each question was presented three times with shuffled options, and majority voting was used to reduce positional bias and assess answer consistency.
- Free-Response Questions (FR): These were graded using an LLM-as-a-judge approach, with DeepSeek-R1:32B acting as the judge on a deterministic 0-5 scale, using a rubric provided by the experts.
They also explored the impact of two augmentation strategies: "+think" (enabling chain-of-thought reasoning) and "+tool" (enabling web search and browsing).
Datasets
The primary dataset introduced in this paper is CAKE, a benchmark for cloud-native software architecture knowledge. It consists of 188 expert-validated questions. The authors also mention using existing resources for concept extraction, such as the AWS Well-Architected Framework, microservices.io, Azure Architecture Patterns, and Kubernetes Enhancement Proposals.
Key Insights and Ideas
- MCQs vs. Free-Response: Multiple-choice questions are insufficient for fully evaluating architectural understanding. LLMs can achieve near-ceiling accuracy on MCQs, but their free-response capabilities reveal significant gaps, especially in design and implementation.
- The "Conviction" Signal: The consistency of answers in MCQs (unanimous vs. split majority) is a strong indicator of accuracy. Unanimous answers are significantly more reliable, offering a practical confidence signal for practitioners.
- Size Matters, But Not Linearly: While larger models generally perform better, the gains in MCQ accuracy plateau significantly above 3 billion parameters. However, free-response performance continues to differentiate models across a much wider parameter range.
- Augmentation Trade-offs: Reasoning augmentation (+think) can improve free-response quality but may negatively impact MCQ performance when models are already near their ceiling, potentially due to "overthinking." Tool augmentation (+tool) is only effective for larger models (around 8B parameters and above), with smaller models showing degraded performance.
- Family-Specific Scaling: The performance scaling patterns within LLM families suggest that architectural knowledge gains might be more dependent on training data composition than solely on parameter count.
Key Results
- MCQ Performance: 16 out of 22 models achieved over 90% accuracy on MCQs. Above 3B parameters, MCQ accuracy gains per parameter step were minimal (1-3%). GPT models achieved near-perfect MCQ accuracy.
- Free-Response Performance: Scores ranged from 1.71 to 4.52. Implementation-level free-response questions were the most differentiating, with scores ranging from 1.36 to 4.54. Mistral models consistently ranked highest among local models for free-response.
- Augmentation Effects: +think generally improved free-response quality (+0.15 to +0.42 FR overall). For small models (e.g., Llama3.2 1B), +tool significantly degraded both MCQ accuracy and free-response scores.
- Mistral's Efficiency: Mistral models showed strong free-response performance, with Mistral 14B offering a good balance of quality and latency compared to larger GPT models.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope: CAKE focuses on cloud-native architecture and may not transfer to classical software patterns.
- MCQ Distractor Quality: Expert review noted some bias in distractors, though option shuffling and the CAKE-Core subset aim to mitigate this.
- Excluded MCQs: 12 implementation-level MCQs were excluded due to formatting issues.
- Judge Model: Free-response evaluation relied on a single primary judge (DeepSeek-R1:32B), with some validation.
- Inter-rater Reliability: Ordinal Krippendorff's alpha values were low, indicating a ceiling effect rather than disagreement, but future work might explore wider rating scales.
- Correct Answer Patterns: A tendency for the longest option to be correct in MCQs could be exploited; future work should control distractor length.
Future work includes incorporating correctly formatted implementation-level MCQs, expanding the scope to classical architectures, investigating optimal augmentation strategies for different model sizes, and extending the benchmark to multilingual settings.
Summary updated