Inside Red Sift

Sifts are implemented as JSON first, Directed Acyclic Graphs, that's a bit of a mouthful, so let's break that down.

• JSON first: Nodes inside a Sift import and export JSON data. This does not mean that all data must be JSON as binary data is appropriately wrapped and XML data is transformed into a representative structure.
• Directed: There is a one-way flow of data between nodes.
• Acyclic: A limitation of the platform is that the nodes in the graph cannot create cycles. This makes it easier for the platform to reason about the graph and very hard for you to create oscillations or other unexpected high order responses.
• Graph: A graph in this context is the set of nodes and the connections between them.

Dagger

Dagger, our stream processing engine, is purpose built for processing a large volume of small events when compared to other graph technologies. The focus on size is evident in the platform limits set by us, but we trade this for lower overhead latencies i.e. single-digit millisecond latencies for complex joins and filters. In many ways, Dagger provides a simpler set of primitives compared to other Stream processing solutions. However, each primitive is much more expressive in Dagger.

In addition, security is a primary design goal with our graph engine. Our process model is unique in that it provides isolation from the underlying platform by data containerisation and isolation. Processes on the graph cannot accidentally or maliciously crosstalk without unless they are specifically allowed to. The data units themselves are either entirely ephemeral or encrypted at rest with per user AES 128-bit keys.

These trade-offs are a fit for the domains we are addressing and map well against email and other message event sources, as each piece of data is typically much smaller than other 'big-data’ domains. Interactive applications demand low, predictable latencies and this is what Dagger delivers.