Software Connectors

Streaming Connector

Basic Concepts

Design Space

Survey

• StreamIt (2002)
• CQL (2003)
• Sawzall (2003)
• Borealis (2005)
• SPC (2006)
• StreamFlex (2007)
• DryadLINQ (2008)
• S4 (2010)
• Storm (2011)
• WebRTC (2011)
• D-Streams (2012)
• TimeStream (2013)

Design Space Overview

Declarative

Select Sum(O.cost)
From Orders O, Fulfillments F [Range 1 Day]
Where O.orderID = F.orderID And F.clerk = "Sue"
    And O.customer = "Joe"

Imperative

void->void pipeline Minimal {
    add IntSource;
    add IntPrinter;
}
void->int filter IntSource {
    int x;
    init { x = 0; }
    work push 1 {
        push(x++); }
    }
int->void filter IntPrinter {
    work pop 1 {
        print(pop());
    }
}

Cluster

Cloud

Pervasive

Web Browser

Static

General topology changes (e.g. operator addition) require to stop and restart the stream

Predefined dynamic behavior can be expressed with optional operators and branches

Dynamic: Operator

Operators can be reconfigured at run-time (e.g. degree of parallelism, resource allocation)

Failed operators can be migrated without altering the logical topology

Dynamic: Topology

Operators can be added (or removed) from the topology without stopping the stream

Load Shedding

Dynamic Initialization

Dynamic Adaptive Controller

Summary

Method

• Decisions harvested from 13 streaming frameworks and languages (2002-2013)
• Intepret Documentation; Download and Assess; Personal Communication
• Simple Architectural Decision Meta-Model (Issue, Alternative, Benefits, Challenges, Examples)

Conclusion

• Design Space for Stream Connectors: 6 design issues and 18 alternatives
• Design time: DAG + Imperative Representation (Textual/Visual)
• Deployment: From Cluster to Cloud, towards the Edge (Pervasive Devices, Web Browser)
• Dynamic Flexibility: From Static to Operator or Topology (orthogonal)
• Run time: From Load Shedding to Decentralized Elastic Pipelines