End-to-End Benchmarking for Edge Computing

12 concepts7 papers1 proposal

Edge computing promises to unleash real-time artificial intelligence and low-latency applications by moving computation closer to data sources. However, evaluating these systems is notoriously difficult. As Mehmet Demir et al. point out in Table of Contents, the absence of standardized benchmarking frameworks makes it nearly impossible to objectively compare different edge platforms or applications. Without reproducible metrics that span the entire application lifecycle, developers cannot effectively evaluate new technologies or optimize system development.

To solve this, a new paradigm of end-to-end edge benchmarking has emerged. Rather than evaluating components in isolation, this approach captures the full computational path across the distributed cloud-edge continuum.

The Fallacy of Isolated Metrics

Traditional performance evaluation in computing often relies on isolated microbenchmarks, such as measuring raw server-side inference speed or peak network bandwidth. In an edge environment, however, these isolated metrics fail to capture the holistic user experience.

Consider an augmented reality (AR) application. As Shilpa Anna George et al. demonstrate with the OpenRTiST benchmarking suite in OpenRTiST: End-to-End Benchmarking for Edge Computing, an AR application combines heavy computation with high bidirectional bandwidth and extreme latency sensitivity. If a benchmark only measures how fast an edge server executes a deep learning model, it completely ignores the time spent capturing data on the mobile device, encoding it, transmitting it over a wireless network, and rendering the result back to the user.

An end-to-end perspective is required because the true bottleneck of an edge system is rarely just the processor; it is the intricate interplay between data acquisition, network transmission, and execution.

Mapping the Multi-Layered Architecture

To capture real-world performance, modern edge benchmarks model applications across a decentralized, multi-layered architecture. For example, Tianshu Hao et al. introduce a framework in Edge AIBench: Towards Comprehensive End-to-End Edge Computing Benchmarking that divides the edge AI pipeline into three distinct layers:

  1. Client-side devices: Sensors, smartphones, or autonomous vehicles that collect data.
  2. Edge computing infrastructure: Local edge servers or gateways that process data with low latency.
  3. Cloud servers: Centralized data centers used for heavy training or long-term storage.
[ Client Devices ] <---> [ Edge Infrastructure ] <---> [ Cloud Servers ]
 (Data Collection)        (Low-Latency Inference)       (Heavy Training/Storage)
       |                            |                           |
       +----------------------------+---------------------------+
                        Full End-to-End Data Path

By evaluating tasks across this entire lifecycle—from data collection and training to validation and inference—benchmarks can uncover how performance degrades or improves when workloads shift between layers. This holistic view is essential for capturing the complex interactions and dependencies inherent in real-world edge AI applications.

Edge systems do not operate in a vacuum of unlimited power and bandwidth. They are defined by severe resource constraints, meaning that optimizing for a single metric like speed often degrades another, like energy efficiency.

As Xiaofei Wang et al. emphasize in Convergence of Edge Computing and Deep Learning: A Comprehensive Survey, unified benchmarks must integrate cross-disciplinary metrics that simultaneously account for networking performance, power consumption, and computational efficiency. When these metrics are evaluated together, hidden architectural trade-offs become visible.

For instance, in EdgeBench: Benchmarking Edge Computing Platforms, Anirban Das 0004 et al. show that while edge computing can reduce bandwidth usage by up to 81x and costs by over 5x compared to cloud-only models, internal architectural choices matter immensely. Choosing between deploying serverless functions or Docker containers on an edge platform creates significant, non-obvious trade-offs in latency and resource consumption that only end-to-end benchmarking can quantify.

Similarly, when evaluating novel hardware architectures, multi-dimensional benchmarking is critical. Li Zhang et al. prototyped a new form of edge server built from massive clusters of mobile Systems-on-Chips (SoCs) in More is Different: Prototyping and Analyzing a New Form of Edge Server with Massive Mobile SoCs. To prove its viability, the authors had to develop a specialized benchmark suite targeting edge-specific workloads like video transcoding and deep learning inference, measuring the precise relationships between throughput, latency, and power consumption to demonstrate the architecture's energy efficiency advantages.

The Future: The Cloud-Edge Continuum

As the field evolves from simple client-server setups toward a highly decentralized "computing continuum," the complexity of benchmarking will only increase. Future applications will deploy Large Language Models (LLMs) and autonomous agents directly to end-user devices, shifting workloads dynamically based on environmental conditions.

In Revisiting Edge AI: Opportunities and Challenges, Tobias Meuser et al. argue that the next generation of benchmarks must be able to evaluate performance across this fluid continuum using multi-objective criteria. Developers will need to simultaneously balance four conflicting pillars:

  • Accuracy: The quality of the AI model's outputs.
  • Latency: The total time elapsed from data generation to action.
  • Energy: The power consumed by heterogeneous, battery-constrained edge devices.
  • Privacy: The security of data as it moves across distributed nodes.

By moving away from isolated component metrics and embracing comprehensive, end-to-end evaluation frameworks, the research community can finally establish the standardized baselines needed to optimize distributed intelligence at scale.

Key Foundational Work

Edge AIBench: Towards Comprehensive End-to-End Edge Computing Benchmarking (2018) — contributed 5 concept elements to this direction.
Table of Contents (2020) — contributed 4 concept elements to this direction.
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_end_to_end_edge_benchmarking_480203d3,
  title = {end-to-end edge benchmarking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-edge-benchmarking/rdr_480203d31c704d2f8cbff48a0bcbd596},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_end_to_end_edge_benchmarking_480203d3_v1,
  title = {end-to-end edge benchmarking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-edge-benchmarking/rdr_480203d31c704d2f8cbff48a0bcbd596/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake