HTTPS

Format

Once an HTTPS webhook is created, you will be returned a fully qualified URI Template that you may use to call your webhook. URI Templates are formalised via RFC6570 and expansion libraries are available in most languages. We currently support Level 4 expressions.

When using the template, your Sift should only mutate the parts of the URI that have been templated as the request is signed to prevent arbitrary tampering.

Pipeline

The pipeline for HTTPS webhooks is a follows:

1. Request URL + specified URI template generates URI key/values
2. (optional) URI key/values & Server Data generates a JSON intermediate document
3. (optional) JSON equivalent document + specified JSONPath template (see below) generates a bucketed data JSON document

HTTPS webhooks must be created with a URI Template that extracts some data out of the URI as the objective of a webhook is to finally emit data that can be used for further computation.

The simplest webhook that will generate data is as follows:

https://webhooks.redsift.io/hook/sV9v9jhflHUwD5/2pFDBtDFwPGZfoFlOP9jSx5lxP3KskpfMQrW26/s/{key}/{value}

If key and value were replaced with MY_KEY and MY_VALUE this URI template would expand to the following URL:

http://webhooks.redsift.io/hook/sV9v9jhflHUwD5/2pFDBtDFwPGZfoFlOP9jSx5lxP3KskpfMQrW26/s/MY_KEY/MY_VALUE

This would be equivalent to the intermediate JSON document:

{
  "uri": {
    "key": "MY_KEY",
    "value": "MY_VALUE"
  }
}

If no JSONPath template is specified, no further processing takes places and this is transformed trivially to the bucketed data JSON document

{
  "key": "MY_KEY",
  "value": "MY_VALUE"
}

which is conveniently the minimum required as the expected output of a webhook. This data will be emitted against the default data bucket in the context of your Sift.

While this may be enough to trigger a computation of some kind in your Sift, most likely you would wish to perform additional transformations before further computation. You can accomplish this by providing a JSONPath template that lets you assemble a JSON payload in the form that proves useful to your computation efforts.

A JSONPath template can take the intermediate JSON document above, and transform those values and a selection of server provided fields to create any combination of key/value/twoI you need on the back of a discrete HTTP GET request. For the purposes of the JSONPath expansion, you may notionally consider the result of a HTTP GET in the example above to be this expanded intermediate JSON document:

{
  "uri": {
    "key": "MY_KEY",
    "value": "MY_VALUE"
  },
  "server": {
    "epoch": 1439810215,
    "count": 1,
    "who": "guid"
  }
}

📘

When considering this intermediate JSON document, the fields names key and value have no semantic significance and are arbitrarily chosen URI template parameters in this example. This is distinct to the outputted bucketed data JSON document where the fields key and value are formal requirements.

The JSONPath template should take this structure and transform it into the expected bucketed data JSON document structure. E.g. the simplest JSONPath template will just copy the values into the canonical bucketed data JSON document structure.

{
  "key": "$.uri.key",
  "value": "$.uri.value"
}

Our template language combines Stefan Goessner’s JSONPath with a JSON document to provide a basic "XSD for JSON" lite templating language. Note, this implementation is limited by design as logic and truly arbitrary manipulation of the JSON document should ideally be done upstream i.e. the originator of the webhook, or downstream in the computation engine. Refer to the original JSONPath documentation for further details on the language though note the limitations further below.

This simple primitive still allows for more sophisticated behaviour when combined with server data.

Primitive types and Quotes

A JSONPath template should be considered a well formed JSON document with appropriate escaping to ensure the output does not end up being malformed over the expected range of inputs. To this effect, the JSONPath expressions should always be provided in quotes. If the expression evaluates to a primitive type or null, the template expansion will replace it with the appropriate primitive type if possible, otherwise it will be transformed into a string.

For example: assuming count is equal to the number 42, the expression "someindex" : "$.value" will evaluate to "someindex" : 42 but "someindex" : "magic/$.value" will evaluate to "someindex" : "magic/42". Note that uri elements are always matched as strings.

🚧

Spaces

Do not include spaces in your JSONPath expression as that is a reserved character and is optionally used to delimit an expression.

Server data

As you may have noticed in the Pipeline section's example document above, the server injects a few fields into the document you may wish to use in your persisted data structure. Currently, the following are provided by default.

📘

By default, these fields are discarded and you must explicitly wire them into your data if you wish to make use of them. E.g., creating a new key monotonically for every invocation of the webhook that includes the best guess user as part of the value.

{
  "key": "$.uri.key/$.server.count",
  "value": {
    "url-value": "$.uri.value",
    "who-was-it": "$.server.who"
  }
}

Use of 'who’

At Red Sift, we take our user’s privacy seriously. At the same time, we acknowledge that non personally identifiable IDs make life easier and reduce the friction of using services and tools when the user permits their use. With the who field, we provide Sift developers some functionality that allows them to safely perform limited disambiguation of anonymous users. It is populated with a likely value for a user that allows a developer to distinguish a specific user/browser from another but is meaningless beyond a hashed string. This value is also salted with the Sift and Account ID and hence can only be correlated in a specific Sift installation. If a user’s privacy settings are set to be non-permissive, you can expect this ID to cycle on every invocation in keeping with the user’s intent. Best case, this ID will timeout and cycle after 1 week so it is best used for short term disambiguation. The properties of this has ensure a vanishingly small probability of collision so you may reliably accept invocations with the same ID to correspond with the same user/browser instance.

Request Data

Payload properties from webhooks can be accessed under $ dollar accessor and injected directly in the definition of your JSONPath. e.g. $.uri.param1 or $.body.param1

📘

URI and query components

Parameters in the URI and query components need to be URL Encoded (Percent Encoded) e.g. method in Javascript is called encodeURIComponent()