Data feeds

We support publishing data feeds to an Amazon S3 bucket or SFTP endpoint. This allows you to export contacts as they enter (or exit) a flow, and record the occurrence of other events. The file path, type, naming, schema, control protocol, and optional conditions for Simon data feeds are broken out below.

Usage

  1. When setting up a new flow, scroll to the bottom of the page to section titled "Configure Data Feed (Optional).
  2. Select "Yes".
  3. Fill in the setup fields and save or launch the flow.
990

Feed configuration

Data format

Select the format of the data export from the following options:

  • CSV
  • TSV
  • JSONL

(Optional) Hash identity

Optionally, select the checkbox if you want a contact's identity information to be hashed using a SHA-256.

Data target

Select S3 or FTP.

Host/bucket

Enter the name of the S3 bucket or SFTP endpoint you wish to write to. Do not include the scheme (e.g. S3://) or a trailing slash (/) in the name.

Path

Enter the file path you wish to write to. Do not include a beginning or trailing slash (/) in the name. Files are uploaded to the following path: /path/year/month/day.

Personalized content

Add fields to the data feed in addition to the default values in the schema. Add a name and value using the same steps to setup Custom Contexts.

Feed format

File types

Two types of files are produced: data files and control files.

File name

Data files are named by the client_flow_operation_random_YmdHMS with an extension that indicates the file type (either ".csv" or ".jsonl").

Data file schema

The file will include, for each contact, one entry for the data feed and one for each applicable flow action.

ParameterDescription
identifierEmail Address
timestampTime of export, with format: Y-M-D HH:MM:SS (e.g. "2017-03-14 17:54:13")
segment_nameName of the segment
flow_nameName of the flow
actionFlow action associated with the entry
variantExperiment variant (null if there is not one)
operationType of operation performed (either 'add' or 'remove')
Custom Contexts (optional)

Control files

Control files are named identically to their corresponding data file with the prefix control_ and are always JSON files. They contain a single JSON object with the following structure:

ParameterDescription
deltaReserved for future use; always true
file_countReserved for future use; always 1
filesA single-element array of metadata; see below

The files array will contain a single JSON object that provides information about the corresponding data file. It contains:

bucketThe specified bucket
pathThe specified path
file_typeEither "csv" or "jsonl"
fileFull URL of data file
rowsThe number of rows in the data file
flow_IDThe ID of the flow
flow_nameThe name of the flow

Currently this produces one control file for every data file, but in the future it may produce a single control file for many data files.