Performance profiling with EndoScope, an acquisitional software monitoring framework

Alvin Cheung, Samuel Madden

TL;DR

This paper introduces EndoScope, a novel software monitoring framework that allows users to pose declarative queries about program behavior. Unlike traditional profilers, EndoScope is "acquisitional," meaning it only instruments the parts of a program necessary to answer specific queries, significantly reducing overhead. This approach enables more flexible and efficient performance analysis, especially for complex systems.

What Problem Does This Paper Address?

Understanding and monitoring the performance of large, complex software systems is a significant challenge. Existing profiling tools often impose a substantial overhead, slowing down the very programs they aim to analyze. Furthermore, these tools typically offer limited functionality, focusing on broad metrics like total function execution time or CPU usage. This leaves a gap for developers and administrators who need to monitor specific aspects of their applications in real-time, such as detecting when a particular query exceeds a service level agreement (SLA), identifying security breaches, or pinpointing performance bottlenecks for dynamic recompilation. The authors aim to bridge this gap by providing a flexible, query-driven framework that minimizes performance impact.

Methods

EndoScope's core innovation lies in its acquisitional and declarative approach.

  • Acquisitional Monitoring: Instead of instrumenting the entire program upfront, EndoScope only instruments the specific parts of the code or system resources that are required to answer a user's query. This is a key differentiator from traditional, non-acquisitional profilers.
  • Declarative Query Language: Users express their monitoring needs using an SQL-like declarative language. This allows them to specify what they want to monitor, rather than how to monitor it.
  • Query Optimization: The framework employs a suite of optimizations, drawing inspiration from database query optimization techniques. This includes control flow graph analysis and predicate pushdown, adapted for the context of program instrumentation. The goal is to find the most efficient physical instantiation of a query, minimizing instrumentation points and overhead.
  • Data Streams: Program state and performance metrics are modeled as data streams. Users can query these streams, and the results of queries are themselves data streams, enabling chaining of analyses.
  • Architecture: EndoScope has a two-part architecture: a server component running on the program execution site and a client component on the monitoring machine. This allows for remote monitoring and flexible placement of query processing.

Datasets

The paper evaluates EndoScope using the following:

  • SimpleApp: A test program included with Apache Derby.
  • TPC-C-like benchmark: A Java implementation of the TPC-C benchmark with specific configurations for warehouses, districts, customers, and orders.
  • Petstore reference application: A Java application hosted on Apache Tomcat.

The authors did not introduce new benchmarks but used these existing, representative applications to demonstrate the effectiveness of their framework.

Key Insights and Ideas

  • Acquisition is Key to Low Overhead: The most significant insight is that by being "acquisitional" (only instrumenting what's needed), EndoScope can achieve dramatically lower overheads than traditional profilers, especially when monitoring only a small subset of program behavior.
  • Declarative Queries Simplify Complex Monitoring: The SQL-like query language empowers users to express sophisticated monitoring requirements without needing to delve into low-level instrumentation details. This democratizes advanced performance analysis.
  • Database Optimization Techniques are Applicable: The paper demonstrates that traditional database query optimization strategies can be effectively adapted to the problem of program instrumentation and query plan generation, leading to more efficient monitoring.
  • Flexibility Beyond Profiling: EndoScope's design allows it to be used for a wider range of applications beyond traditional profiling, including debugging, alerting, and real-time anomaly detection.

Key Results

  • Lower Overhead: In profiling the Derby Database running TPC-C, EndoScope's overhead ranged from 1% to about 25%, while the fastest existing profiler measured imposed a minimum overhead of about 30%.
  • Scalability: The number of tuples generated by EndoScope scales approximately linearly with the percentage of program monitored, demonstrating predictable behavior.
  • Effectiveness of Optimizations: Experiments showed that different query plan strategies (e.g., where to perform aggregation – client vs. server) have a significant impact on performance, and an adaptive approach is beneficial.
  • Comparison with Existing Tools: EndoScope's profiler implementation showed comparable or better performance than tools like visualvm, jip, jrat, and hprof, especially when monitoring a small fraction of methods.

Limitations and Future Work

  • Plan Optimizer Implementation: The full plan optimizer was not implemented in the prototype; hand-optimization was used for evaluation. Developing a robust, adaptive plan optimizer is a key area for future work.
  • Cost Model Complexity: Accurately estimating the cost of different query plans, especially considering dynamic program behavior and trigger execution, is challenging and requires further research.
  • Approximate Query Answers: The current system focuses on exact answers. Exploring approximate query answers, similar to sampling-based profilers, could offer further optimization opportunities.
  • API for System Monitoring: While EndoScope provides a query interface, a more library-like API could be beneficial for tool developers building on top of the framework.
  • Distributed Systems: Extending EndoScope to monitor distributed applications is a significant future direction.

Summary updated

BibTeX
@article{cheung2008performance,
  author = {Alvin Cheung and Samuel Madden},
  title = {{Performance profiling with EndoScope, an acquisitional software monitoring framework}},
  journal = {Proc. VLDB Endow.},
  year = {2008},
  url = {https://openreview.net/forum?id=lj_EnOFaflr},
}
Figure