Streaming Across the Web of Things

Goals

1. Provide a stream processing framework for JavaScript
2. Deploy streaming pipelines across the Web
   (on Web-enabled devices, Web servers and Web browsers)

Why JavaScript?

1. Low-level streaming protocols (WebSockets, WebRTC)
2. Dynamically typed streams (send/receive JSON)
3. Actor-like parallelism (HTML5 WebWorkers, Node.js cluster)
4. Shared-memory parallelism (TigerQuoll)
5. Runs on almost everything

Operators

var k = require('WLS.js');
k.createOperator(function(msg) { 
  //operator function
  var chunks = msg.data.split(’ ’);
  chunks.map(function(v) {
    //output to the stream
    k.send({ word : v });
  });
}).start();

• The operator function is called for every incoming stream element

Producer

var k = require('WLS.js');

function gps(position) {
    k.send(
     { lat : position.coords.latitude,
       lon : position.coords.longitude });
}

setInterval(function(){
//HTML5 API in the Web Browser
  navigator.geolocation.getCurrentPosition(gps);
}, 1000);

Stateful Filter

var k = require('WLS.js');

//initialize operator state
var sum = 0;
var samples = 0;

k.createOperator(function(msg) {
  //handle next stream message

  sum += msg.temperature;
  samples ++;
  
  //check average temperature
  if (sum/samples > 25) {
    k.send({'weather' : 'warm'});
  } else {
    k.send({'weather' : 'cold'});
  }

}).start();

UI Consumer

var k = require('WLS.js');

k.createOperator(function(msg) {

  //handle weather condition message
  var w = msg.weather;
  if (w) {
    k.setDOM('#weather', 'Temperature is: ' + w)
  }
  
  //handle gps location message
  var lat = msg.lat;
  var lon = msg.lon;
  if (lat && lon) {
    k.setDOM('#position', 
         'Position is: ' + lat + "," + lon);
  }
  
}).start();

Stream Topologies

Many ways to configure topologies (operator graphs):

• Interactive Console: start/stop operators and bind them together
• Predefined topology: load a topology.json as initial configuration
• API: programmatically create and modify the topology graph while it is running

Interactive Console

run op.js

Run an operator somewhere (returns the operator ID)

bind op1 op2

Establish a stream connection between two operators

stop op1

unbind op1 op2

update op1 script op.js

update op1 worker N

Topology Example (JSON)

{
  "topology" : {
    "id" : "example-topology",
    
    "operators" : [

      {
        "id" : "producer",
        "script" : "webcam.js",
        "deploy" : {
          "peer" : "IP:port",
          "path" : "/webcam",
          "sensors" : ["webcam"]
        }
      }, {
        "id" : "filter",
        "script" : "filter.js"
      }, {
        "id" : "consumer",
        "script" : "consumer.js",
        "deploy" : {
          "browser" : true,
          "path" : "/consumer"
        }
      }
    ],
    
    "bindings" : [

      { 
        "from" : "producer", 
        "to" : "filter",
        "type" : "roundrobin"
      }, {
        "from" : "filter",
        "to" : "consumer",
        "type" : "broadcast"
      }
    ]
  }
}

Deployment Constraints

peer : [ "IP:port" ]

The operator can be deployed on these peers

sensors : [ "webcam", "gps" ]
actuators : [ "light-switch" ]

browser : undefined

browser : false

browser : true

path : "/webcam"

Next Steps

• More expressive pipeline (join synchronization, replicated stateful operators)
• Large Scale Deployment, experiments on Tessel devices
• Optimized Pipeline deployment (controller, network ping latency, peer capacity benchmarking)

References

• M. Babazadeh, C. Pautasso, A RESTful API for Controlling Dynamic Streaming Topologies, WS-REST 2014 at WWW 2014
• M. Babazadeh, C. Pautasso, The Stream Software Connector Design Space: Frameworks and Languages for Distributed Stream Processing, WICSA 2014
• D. Bonetta, W. Binder, C. Pautasso, Tiger Quoll: Parallel Event-based JavaScript, PPoPP 2013
• D. Bonetta, D. Ansaloni, A. Peternier, C. Pautasso, W. Binder, Node.Scala: Implicit Parallel Programming for High-Performance Web Services, Europar 2012
• B. Biörnstad, C. Pautasso, G. Alonso, Control the Flow: How to Safely Compose Streaming Services into Business Processes, Proc. of the 2006 IEEE International Conference on Services Computing, Chicago, USA, IEEE, pp. 206 - 213, 2006.