Implementation

Dagger gives node implementations 30 seconds to respond per invocation, later responses are discarded. For long compute nodes this period is extended to 30 minutes (needs to be requested as a QoS requirement in sift.json).

Node input

The input object provided is a dictionary containing up to four fields:

{
  "input": {
    "in": {},
    "query": [],
    "with": {},
    "get": []
  }
}

📘

Java, Scala and Clojure

For Java, Scala and Clojure the ComputeRequest object will be passed to the node instead.

{
  "bucket": "bucket_name",
  "data": []
}

[{
  "bucket": "bucket_name",
  "key": "key_name",
  "data": [{
    "key": "key_name",
    "value": <bytes>
  }]
}]

Data

Self evidently, this is the smallest quantum of data an input can source or sink and is also the smallest unit of work a node can operate on. Data contains:

"data": {
  "key": "key1",
  "value": [], // byte array
  "epoch": 1457700316
}

🚧

Keys

Keys themselves are not encrypted and when creating your own keys, take care when designing your Sift so as not to accidentally expose PII in this field.

📘

key and ‘key$schema’ field limitations

• only UTF-8 strings
• slash / separator of hierarchy
• up to 7 levels in a hierarchy

📘

'value' field

This field is encoded as a byte array internally and is accessible in the respective node implementation like so:

Javascript: Buffer
Python 2.7: str
Python 3: bytes
Java, Scala, Clojure: byte[]
Swift: Data?
Go: []byte

You will then need to convert this back to the right representation you need.

BucketedData

In most instances, data is wrapped as BucketedData which adds one or multiple of the following fields to the data:

Node outputs

An implementation can return nothing, i.e. undefined or null in Javascript, if it wishes to do nothing with the data.

If it returns any data at all, a minimum requirement for an implementation is to return a dictionary with a key field. If no value is provided for this key, the value is assumed to be empty. The third field for a key is the name of the store it is meant to be stored in.

"datum": {
  "name" : "bucket1", //name of bucket
  "key": "key1", // only string
  "value" : {}, // string OR {} OR [] OR Byte array
  "epoch" : 1457976598 // optional
}

Allowed outputs for a node:

📘

Javascript & Promises

A node implemented in JavaScript can also return a Promise or an array of them but each Promise should resolve to the dictionary mentioned above.

📘

'name' field

If more than one store is defined as an output for a node, the name property is required to avoid errors. If there is only one store it can be omitted although it is recommended to always add it for completeness.

Node outputs for Thread and Thread List Views

An implementation can emit to the special buckets _email.id and _email.tid the email or thread specific data that it needs the same way we have described until now. The only difference is when we want to target data towards the email-thread or the thread-list view, then we need to output data under the detail or the list attribute respectively.

• All the data under the detail attribute will become available in the Sift Controller under the state.params.detail key as described in Sift Controller
• All the data under the list attribute will become available in the listInfo argument as described in Email Client Controller

datum = {
  "key": "key1",
  "value": {
    "detail": {},
    "list": {}
  }
}

Environment Variables

Things like secrets and configuration options can be passed to the node implementations through the usage of environment variables. Those can be defined through the Sift Control Panel in the SDK, the cog icon on the left, which looks like this:

📘

Workshop Sifts

For everything that's relevant for Sifts running in the Workshop we will have to authenticate you first. e.g. The screen above is after I authenticated with my account.

First section is for the Sift you are developing locally in this case the TL;DR, next section is for the Sifts you have deployed in your Workshop. You can access the secrets menu through the keyhole icon on the relevant Sift tile. After clicking it you should see a screen similar to this:

Here you can add all the environment variables you would like to pass to your nodes and then in the implementation of a node you can access them with the standard way your language provides for accessing environment variables. e.g. in Javascript all of them would be under process.env

❗️

Naming restriction

Variable names can not start with an underscore _ the character is reserved for internal use.

🚧

Environment variables and Workshop Sifts

Variables that you define for Workshop Sifts are branch specific. So for example variables defined on the master branch will not be available on the development branch.

Remote Logs

When a Sift is running on our cloud you can subscribe on the logs that it produces via the Sift Control Panel in our SDK. Just head to the Control Panel the same way we described in the section above and look for the icon that looks like a sheet of paper in the relevant Sift tile. It's next to the keyhole one you can not miss it! After clicking it you should see a screen like this:

📘

Logs and timing

You need to navigate to this page (subscribe for logs) before actually doing anything on our cloud (like installing the sift) otherwise you will miss them.