TL;DR
This paper introduces Log², a novel logging mechanism designed to balance performance logging effectiveness with system overhead. By intelligently filtering logs based on a configurable budget, Log² significantly reduces the volume of logs without sacrificing the ability to diagnose performance issues, making it practical for large-scale online services.
What Problem Does This Paper Address?
In modern large-scale online service systems, logging is crucial for monitoring and diagnosing performance problems. However, intensive logging introduces significant overhead, including increased disk I/O, CPU, and memory consumption, which can even degrade the system's performance. Furthermore, the sheer volume of logs generated often leads to a large number of redundant or "useless" entries, making it challenging and time-consuming to pinpoint actual performance issues. Existing methods for reducing logging overhead, such as manual log removal or sampling, often compromise logging effectiveness or lack fine-grained control. This paper addresses the need for a logging system that can dynamically control its overhead while maximizing its diagnostic utility.
Methods
Log² employs a two-phase filtering mechanism to achieve its cost-aware logging.
- Two-Phase Filtering:
- Local Filter: Each thread has a local filter that calculates a "utility score" for each logging request. This score quantifies how useful a log is for performance diagnosis. Logs with utility scores below a dynamically adjusted global threshold are discarded immediately.
- Global Filter: Logs that pass the local filter are sent to a global filter. Here, logs are cached, and a periodic process sorts them by utility score. Only the top-ranked logs, up to a predefined logging budget (defined as logging bandwidth, e.g., KB/second), are flushed to disk. This ensures that the system stays within its budget.
- Utility Score Calculation: The utility score is based on the deviation of a Monitored Code Region's (MCR) execution time from its historical behavior. The paper proposes three formulas for utility scores, primarily using the mean ($\mu$) and standard deviation ($\sigma$) of execution times. The system dynamically updates these statistics using exponential smoothing.
- Dynamic Threshold Adjustment: The global threshold for the local filters is adjusted dynamically based on the volume of logs cached in the global filter's buffer relative to the budget. This iterative adjustment mechanism (inspired by the Secant Method) aims to quickly converge on a threshold that keeps the log volume within budget while maximizing effectiveness.
- Buffer Design: A swap buffer mechanism is used to decouple the fast insertion of logs by local filters from the slower sorting and flushing operations of the global filter, minimizing contention.
Datasets
The paper evaluates Log² on two main subjects:
- BlogEngine: A popular open-source ASP.NET blogging platform. The authors instrumented ~1000 Monitored Code Regions (MCRs) within BlogEngine.
- Microsoft ServiceX: A large-scale, real-world online service system from Microsoft. The evaluation used 12 datasets, each representing one hour of continuous logging from the system.
No new benchmarks were introduced; instead, existing systems were used to demonstrate Log²'s effectiveness.
Key Insights and Ideas
- The "Whether to Log" Problem: The paper highlights that the critical decision isn't just where to log, but whether to log a specific event, especially when dealing with massive log volumes and tight resource budgets.
- Utility Score as a Proxy for Diagnostic Value: The concept of a "utility score" is central, providing a quantifiable measure of a log's importance for performance diagnosis. This allows for intelligent prioritization.
- Dynamic Budgeting is Key: The system's ability to dynamically adjust its logging behavior based on a configurable budget (logging bandwidth) is crucial for adapting to varying system loads and resource constraints.
- Two-Phase Filtering for Scalability: Separating local filtering from global filtering prevents a single bottleneck and allows the system to handle a high volume of incoming logging requests efficiently.
- Overhead Reduction Beyond Logging: The paper shows that by reducing the number of logs processed, Log² can actually lead to reduced CPU usage compared to a baseline system, as it avoids expensive log formatting and processing for discarded logs.
Key Results
- Logging Throughput Reduction: Log² achieved an average reduction of over 97% in logging throughput compared to a baseline approach that outputs all logs. For example, in the BlogEngine experiment, Log² flushed an average of 104 logs/interval, while the baseline flushed 3,800 logs/interval, all while staying within a budget of 120 logs/interval.
- High Logging Effectiveness: Log² demonstrated high logging effectiveness, achieving nearly 100% coverage of injected performance issues even with significantly reduced log volumes. For instance, in the ServiceX case study, Log² achieved 98% coverage of performance hotspots with a budget of 100 logs/interval, while only recording 4.5 MB of logs compared to the original 500 MB per dataset.
- Low Additional Overhead: The additional memory usage of Log² was negligible. In fact, the integrated system with Log² showed slightly lower CPU usage than the baseline, attributed to avoiding unnecessary log processing.
- Comparison with Sampling: Log² significantly outperformed sampling-based approaches in terms of effectiveness for a given budget. With a budget equivalent to a 3% sampling rate, Log² achieved ~100% coverage, while sampling methods achieved only 2-6%.
Limitations and Future Work
- Single Budget Objective: Currently, Log² primarily focuses on controlling logging bandwidth as the budget. The authors suggest extending this to support multiple objectives, such as CPU and memory usage, for multi-objective optimization.
- "Where to Log" Problem: While Log² addresses "whether to log," the paper acknowledges that "where to log" (i.e., automatically identifying MCRs) is another important topic for future work. They propose enabling "conservative logging" where developers can instrument extensively, and Log² handles the filtering.
- Extension to Failure Diagnosis: The current work focuses on performance diagnosis. Future work includes extending Log² to support other types of analysis, such as failure diagnosis, by leveraging different log types.
- Centralized Budget Control: The current implementation is a runtime library for a single service. Future work could involve a centralized process managing budgets for multiple services and enabling dynamic re-allocation.
Summary updated