User GuidesAPI Reference & Developer GuidesInteractive GuidesOperational StatusRelease Notes
Log In To Simon

S3/SFTP For Outbound Use

Overview

Simon supports data sharing via AWS S3 or a SFTP server to share data out from the Simon platform.

This currently includes the following sources:

  • Simon Dataset Exports
  • Simon Segment Exports
  • Simon Data Feeds
  • Third-party event streams
  • Dataset Exports

Encryption

All customer buckets are encrypted using s3 server-side encryption with unique KMS keys for each customer. Refer to Amazon's documentation for more detail.

Simon Dataset Exports

Description

Occasionally, users will want to access the data used in segmentation or content in a raw format outside of the platform. This might also include gathering additional data based on existing datasets. For example, accumulating total monthly metrics based on SimonSignal events could be done with Dataset Exports. Any existing dataset can be configured to share transformed data. See Types of Datasets for more details on datasets.

Timing

Dataset exports sync at most one time per day. These can be shared on either a daily, weekly, or monthly cadence, depending on user preference.

File Formats

Dataset exports are exported as .CSV files with comma-delimited values. If values exported values include commas, quote characters will be used to wrap that value.

Folder structure

All dataset exports can be written to a specific file path for a given S3 bucket or SFTP location. This can be one of two options:

  • A static file path, where the export is overwritten every time a new export is generated
  • A date-foldered file path, where the export is saved for each date the export is generated. For example,
{root_path}/{year}/{month}/{day}/{files}

s3://ORG.simondata.com/export/2020/03/17/export.csv

Data Format

The data is exported in whatever format is specified in the query underlying the dataset. The data itself won’t be converted or transformed beyond light SQL query-level transformations.

Data Limit

We limit exports to a total of 1GB of data out of Simon per day across all of your data via S3/SFTP. If you need more than that, please speak with your Client Solutions Manager.

For each dataset export, the amount of data scanned per query executed must not exceed 10GB.

Segment Exports

Description

After creating a segment, users can inspect segment membership directly by exporting a file to see the contacts included in the segment (see Exporting Segments for mechanics). For example, after creating a segment to see all contacts that opened a specific email in the past week, the user can export the segment to see all contacts related to the segment and any specified contact fields. Any existing segment can be configured to be exported and have the result shared on a regular cadence.

Timing

Segment exports can be shared on either a daily, weekly, or monthly cadence, depending on user preference.

File Formats

Segment exports can be exported as CSV, TSV, or JSONL files. If values being exported include commas, quote characters will be used to wrap that value.

Folder structure

All segment exports can be written to a specific file path for the given S3 bucket or SFTP location. This can be one of two options:

  • A static file path, where the export is overwritten every time a new export is generated
  • A date-foldered file path, where the export is saved for each date the export is generated. For example:
{root_path}/{year}/{month}/{day}/{files}

s3://ORG.simondata.com/export/2020/03/17/export.csv

Data Format

The identities in the segment can be hashed using the SHA-256 algorithm, if desired. This is specified when setting up the segment export. All other data will be exported in whatever format is used in the underlying custom contexts of the segment export.

Data Limit

We limit exports to a total of 1GB of data out of Simon per day across all of your data via S3/SFTP. If you need more than that, please speak with your account manager.

Data Feeds

Description

We support publishing data feeds to an Amazon S3 bucket or SFTP endpoint. This allows you to export contacts as they enter (or exit) a flow, and record the occurrence of other events. See Data Feeds for more details on setup.

Timing

Data feeds are currently exported on a continuous basis.

File Formats

Dataset exports can be exported as CSV, TSV, or JSONL (JSON newline-delimited) files

Folder structure

All data feeds can be written to a specific file path for the given S3 bucket or SFTP location. The pathing structure within that specific file path is a date-foldered structure. For example:

{root_path}/{year}/{month}/{day}/{files}
e.g. s3://ORG.simondata.com/data-feed/2020/03/17/data_feed.json

Data Format

The identities in the data feed can be hashed using the SHA-256 algorithm, if desired. This is specified when setting up the data feed. All other data will be exported in whatever format is used in the underlying dataset. See Data File Schema for specific details on fields that can be exported.

Data Limit

We limit exports to a total of 1GB of data out of Simon per day across all of your data via S3/SFTP. If you need more than that, please speak with your account manager.

Third-Party Event Streams

Description

For third-party event streams that are ingested directly into Simon, users will occasionally want to access the raw event data for use in their own ETL process outside of Simon. For example, using raw SendGrid events to load data into another vendor’s system. Please contact your account manager for more information.

Timing

Event streams can be shared daily for the previous day’s events.

File Formats

Third-party event stream data format will depend on the format of ingested data, but typically is JSONL.

Folder structure

Data ingested into Simon that’s being shared to a client (e.g., SendGrid events) can either be:

Copied with the existing folder structure in place to a given S3 or SFTP file path
ESP events are stored in Simon in a date-foldered structure, and could be maintained when sharing data. For example:

{root_path}/{year}/{month}/{day}/{files}

s3://ORG.simondata.com/events/2020/03/17/events_20200317.json

Written directly to a given S3 or SFTP file path:

{root_path}/{files}

s3://ORG.simondata.com/events/events_20200317.json

Data Format

The data will be exported in whatever format the raw ingested event is sent as. The data itself won’t be converted or transformed in any fashion.

Data Limit

We limit exports to a total of 1GB of data out of Simon per day across all of your data via S3/SFTP. If you need more than that, please speak with your account manager.