Retrieval Supersedes Scale: Efficient Log Level Prediction via Multiplex Socio-Technical Context

Youssef Esseddiq Ouatiti, Mohammed Sayagh, Bram Adams, Ahmed E. HassanarXiv 2026 · arXiv

TL;DR

This paper demonstrates that for predicting log levels (like DEBUG vs. ERROR), providing an LLM with the right "socio-technical" context is far more effective than simply using a larger model. By retrieving examples based on both code functionality and developer ownership, a small 7B model can outperform massive commercial models like GPT-4o, achieving state-of-the-art accuracy while remaining efficient enough for local deployment.

What Problem Does This Paper Address?

Choosing the correct log level is a constant headache for developers. Set it too low (e.g., TRACE), and you risk missing critical info in production; set it too high (e.g., ERROR), and you drown in noise and storage overhead.

While researchers have tried to automate this using Large Language Models (LLMs) via In-Context Learning (ICL), most current approaches just feed the model random examples. This ignores a fundamental reality of software engineering: logging is socio-technical. Different components have different technical needs, and different development teams have their own "dialects" and conventions for what constitutes an "error" versus a "warning."

Methods

The authors propose a retrieval-augmented approach that moves away from random examples toward Ownership-Functionality Aware Retrieval.

Fig. 1: Overview of our Ownership-Functionality Aware retrieval approach for log level prediction.

The core innovation is the use of a multiplex graph to cluster source files. This graph has two layers:

  1. Functionality Layer: Files are connected based on functional similarity, determined by code embeddings (using CodeXEmbed).
  2. Ownership Layer: Files are connected based on shared maintainers, determined by mining Git history to see which developers frequently modify the same files.

By applying a multiplex variant of the Leiden community detection algorithm, the system partitions a project into clusters that are cohesive both technically and socially.

The multiplex clustering process combines technical functionality and social ownership signals

When a developer adds a new log statement, the "Retrieval Router" identifies the relevant cluster and picks the top $k$ examples that are most similar to the target file. These examples are then used to prompt a frozen LLM (like CodeLlama-7B) to predict the appropriate log level.

Datasets

The researchers evaluated their approach across ten popular open-source Java projects, including heavyweights like Hadoop, Elasticsearch, Cassandra, and Kafka. These projects provided a diverse testbed, ranging from 375 to over 25,000 logging statements, allowing the team to test the robustness of their method across different scales and logging cultures.

Key Insights and Ideas

  • Context > Parameters: A "smart" 7B model with precise context can "punch above its weight," defeating general-purpose models that are 20x to 100x larger.
  • The Social Signal Matters: Code functionality is the primary signal, but developer ownership acts as a crucial secondary refinement. Knowing who writes the code helps the LLM align with local team norms.
  • Diminishing Returns of Scale: Scaling from a 7B to a 34B model yielded only a marginal 0.9% improvement in accuracy but increased operational costs by a staggering 322%.
  • Local is Better: For high-volume workflows like CI/CD pipelines, small, self-hosted models offer the best balance of performance, cost-efficiency, and data privacy.

Key Results

  • State-of-the-Art Performance: The multiplex retrieval method achieved median AUC scores between 0.83 and 0.97 across all projects.
  • Significant Gains: It outperformed random retrieval baselines by a median of 0.07 AUC and crushed previous specialized supervised models (like DeepLV) by a median of 0.12 to 0.19 AUC.
  • Efficiency Wins: A multiplex-enhanced CodeLlama-7B (Avg AUC 0.89) outperformed GPT-4o (Avg AUC 0.79) and DeepSeek-V3.2 (Avg AUC 0.73) in zero-shot settings.
  • Real-time Speed: The retrieval step is highly optimized, with a median latency of just 8.9 ms, making it suitable for integration into IDEs.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Language Scope: The study focused exclusively on Java projects. While the principles should generalize, the specific embedding models and parsers would need to be adapted for other languages.
  • Ownership Dynamics: The ownership signal was derived from voluntary open-source contributions. In corporate environments, where ownership is often assigned by policy rather than organic contribution, the signal might behave differently.
  • Untapped Signals: Future work could incorporate additional socio-technical signals, such as defect density, call-graph proximity, or module hierarchy, to further refine the retrieval process.
  • The "Ground Truth" Problem: The study assumes that the log levels currently in the code are the "correct" ones. Future research could involve human-in-the-loop studies to see if LLM recommendations actually improve debugging efficiency for developers.

Summary updated

Abstract

Developers insert logging statements in source code to record its execution, and the resulting logs are essential for detecting and troubleshooting system anomalies. A key part of these statements is the log level (e.g., DEBUG, INFO, WARN, ERROR), which determines whether a message is recorded. Choosing it is a trade-off: levels set too low mask important information in production, while levels set too high create noisy logs and extra I/O and storage overhead. Recent work uses Large Language Models (LLMs) with In-Context Learning (ICL) to predict log levels, but typically augments the prompt with randomly sampled examples. We argue that this overlooks the socio-technical nature of logging, where conventions vary by component and development team. Off-the-shelf LLMs using random examples are reported to achieve median AUCs of only 0.56 to 0.76, leaving room for improvement without fine-tuning. In this paper, we propose a retrieval approach for LLM-powered log level prediction that prioritizes context relevance over random selection and basic similarity. Our method retrieves logging examples relevant along two socio-technical dimensions: (i) functionality (code performing similar operations) and (ii) ownership (code maintained by the same developers). Evaluating four LLMs across ten popular Java projects, we find that this retrieval improves AUC over random baselines by 0.07 (median), and combining both signals yields median AUCs of 0.83 to 0.97. Our ablation studies show that each signal improves on random retrieval, and that combining them outperforms the stronger of the two on all ten projects. Our results also significantly exceed state-of-the-art supervised predictors (e.g., DeepLV) by a median AUC of 0.12 to 0.19. Our findings highlight the importance of socio-technical signals such as code functionality and ownership for LLM-powered log level prediction.
BibTeX
@misc{ouatiti2026retrieval,
  author = {Youssef Esseddiq Ouatiti and Mohammed Sayagh and Bram Adams and Ahmed E. Hassan},
  title = {{Retrieval Supersedes Scale: Efficient Log Level Prediction via Multiplex Socio-Technical Context}},
  year = {2026},
  eprint = {2611.05901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05901},
}
Figure