property source is s3 Schema #

undefined#/properties/postProcess/items/allOf/21/properties/params/properties/dataSource/oneOf/1/allOf/0
Abstract Extensible Status Identifiable Custom Properties Additional Properties Access Restrictions Defined In
Can be instantiated No Unknown status No Forbidden Allowed none specSchema.schema.json*

0 Type #

unknown (property source is s3)

property source is s3 Properties #

Property Type Required Nullable Defined by
awsAccess object Optional cannot be null specSchema (Unified Pipeline Specification)
timePrefix Flexible Optional cannot be null specSchema (Unified Pipeline Specification)
bucket string Optional cannot be null specSchema (Unified Pipeline Specification)
namespace string Optional cannot be null specSchema (Unified Pipeline Specification)
s3ListObjectsAll boolean Optional cannot be null specSchema (Unified Pipeline Specification)
objectTimeRange string Optional cannot be null specSchema (Unified Pipeline Specification)
s3Sql string Optional cannot be null specSchema (Unified Pipeline Specification)
suffixFilter string Optional cannot be null specSchema (Unified Pipeline Specification)
maxObjects number Optional cannot be null specSchema (Unified Pipeline Specification)
maxObjectsDirection Not specified Optional cannot be null specSchema (Unified Pipeline Specification)
style Not specified Optional cannot be null specSchema (Unified Pipeline Specification)
extractTime string Optional cannot be null specSchema (Unified Pipeline Specification)

awsAccess #

AWS Access Credentials

awsAccess

timePrefix #

timePrefix TODO

timePrefix

bucket #

Top level S3 bucket under which to find data, metadata, or namespace folders

bucket

namespace #

Namespace separation of data files. Folder path under top level S3 bucket in which to find related data files

namespace

s3ListObjectsAll #

s3ListObjectsAll TODO

s3ListObjectsAll

objectTimeRange #

objectTimeRange TODO

objectTimeRange

s3Sql #

S3 SQL statement to select rows of data to be returned from the files in the dataset

s3Sql

suffixFilter #

Only process objects whose key ends with the suffixFilter

suffixFilter

maxObjects #

maxObjects TODO

maxObjects

maxObjectsDirection #

Enumeration of strings with constraints below

maxObjectsDirection

maxObjectsDirection Constraints #

enum: the value of this property must be equal to one of the following values:

Value Explanation
"left"
"right"

style #

Enumeration of strings with constraints below

style

style Constraints #

enum: the value of this property must be equal to one of the following values:

Value Explanation
"conventional" Line delimitted JSON, gzipped. Filename ending: .jsonl.gz
"kinesis-firehose" Line delimitted JSON, uncompressed text. Filename ending: .gz

extractTime #

JSONata expression to extract time from S3 object key, used by objectTimeRange filtering

extractTime