The benefits of service choreography for data-intensive computing

Adam Barker, Paolo Besana, David Robertson, Jon B. WeissmanCLADE@HPDC 2009 · DOI

TL;DR

This paper argues that traditional service orchestration struggles with data-intensive workflows due to scalability bottlenecks. It proposes service choreography as a more efficient, decentralized alternative, introducing a new lightweight calculus (LCC) and framework (OpenKnowledge) to demonstrate its practical benefits in reducing data transfer.

What Problem Does This Paper Address?

Modern computing workflows, especially in scientific domains, are increasingly dealing with massive datasets. Centralized orchestration, where a single engine manages all service interactions and data flow, becomes a significant bottleneck. This leads to excessive data transfer, wasted bandwidth, and performance degradation as the central engine struggles to keep up. The paper aims to address this scalability issue by advocating for and demonstrating the advantages of service choreography, a decentralized approach where services coordinate directly with each other.

Methods

The paper's core contribution lies in its advocacy for and demonstration of service choreography. The authors introduce two key elements:

  • Light Weight Coordination Calculus (LCC): A new, lightweight process calculus designed to formally define choreographies. LCC is based on π-calculus, providing a formal semantic foundation. It allows for the definition of interaction models where peers take on specific roles and communicate directly, without a central coordinator.
  • OpenKnowledge Framework: A practical, choreography-based architecture that implements LCC. This framework enables peers to publish and enact interaction models in an open peer-to-peer system. It handles the discovery of services, subscription to roles, and the dynamic bootstrapping of interactions.

To illustrate the benefits, the paper analyzes common workflow patterns (fan-in, fan-out, sequence) and compares orchestrated vs. choreographed data flow using simple equations and diagrams.

Datasets

Not discussed in the paper. The paper uses hypothetical data sizes (in MB) for illustrative purposes within workflow pattern examples, rather than real-world datasets.

Key Insights and Ideas

  • Choreography as a Scalability Solution: The central argument is that for data-intensive computing, choreography's decentralized nature inherently avoids the bottlenecks of centralized orchestration, leading to significant improvements in data flow efficiency.
  • Data Flow Optimization is Crucial: The paper emphasizes that the primary benefit of choreography in this context is the drastic reduction in intermediate data transfer. Data goes directly where it's needed, bypassing a central hub.
  • LCC and OpenKnowledge: A Practical Approach: The introduction of LCC and OpenKnowledge provides a concrete, formal, and implementable solution for achieving service choreography, moving beyond theoretical discussions.
  • Peer-to-Peer Collaboration is Key: The OpenKnowledge framework highlights the power of a peer-to-peer model where services act as equal participants, each knowing their role and interacting directly.
  • Design-Time vs. Run-Time Flexibility: The paper suggests that LCC and OpenKnowledge offer more dynamic and flexible choreography definitions compared to some other approaches that might require more rigid design-time configuration.

Key Results

The paper demonstrates through workflow pattern analysis that choreography significantly reduces data flow compared to orchestration. For instance:

  • Fan-in Pattern: Orchestrated data flow was 630MB, while choreographed was 330MB.
  • Fan-out Pattern: Orchestrated data flow was 430MB, while choreographed was 330MB.
  • Sequence Pattern: Orchestrated data flow was 362.5MB, while choreographed was 187.5MB.

The authors also note that the benefits of choreography increase with the number of services and data volume, particularly with a higher number of sources. Figure 2 visually represents this increasing difference in data flow.

Limitations and Future Work

  • Hand-Encoding of Protocols: The authors acknowledge that hand-encoding protocols into LCC is time-consuming.
  • Future Work:
    • Developing graphical tools for constructing LCC protocols.
  • Exploring automatic generation of protocols through planning.
  • Conducting a full pattern analysis of LCC based on Service Interaction Patterns.
  • Implementing other workflow patterns (fan-out, sequence) to further validate the approach.

Summary updated

BibTeX
@inproceedings{barker2009benefits,
  author = {Adam Barker and Paolo Besana and David Robertson and Jon B. Weissman},
  title = {{The benefits of service choreography for data-intensive computing}},
  booktitle = {CLADE@HPDC},
  year = {2009},
  doi = {10.1145/1552315.1552317},
}
Figure