property source is s3 Schema #

undefined#/properties/postProcess/items/allOf/24/properties/params/properties/pipeline/oneOf/1/properties/dataSource/allOf/0

Abstract	Extensible	Status	Identifiable	Custom Properties	Additional Properties	Access Restrictions	Defined In
Can be instantiated	No	Unknown status	No	Forbidden	Allowed	none	specSchema.schema.json*

0 Type #

unknown (property source is s3)

property source is s3 Properties #

Property	Type	Required	Nullable	Defined by
awsAccess	`object`	Optional	cannot be null	specSchema (Unified Pipeline Specification)
timePrefix	Flexible	Optional	cannot be null	specSchema (Unified Pipeline Specification)
bucket	`string`	Optional	cannot be null	specSchema (Unified Pipeline Specification)
namespace	`string`	Optional	cannot be null	specSchema (Unified Pipeline Specification)
s3ListObjectsAll	`boolean`	Optional	cannot be null	specSchema (Unified Pipeline Specification)
objectTimeRange	`string`	Optional	cannot be null	specSchema (Unified Pipeline Specification)
s3Sql	`string`	Optional	cannot be null	specSchema (Unified Pipeline Specification)
suffixFilter	`string`	Optional	cannot be null	specSchema (Unified Pipeline Specification)
maxObjects	`number`	Optional	cannot be null	specSchema (Unified Pipeline Specification)
maxObjectsDirection	Not specified	Optional	cannot be null	specSchema (Unified Pipeline Specification)
style	Not specified	Optional	cannot be null	specSchema (Unified Pipeline Specification)
extractTime	`string`	Optional	cannot be null	specSchema (Unified Pipeline Specification)

awsAccess #

AWS Access Credentials

awsAccess

is optional
type: object (awsAccess)
cannot be null
defined in: awsAccess

timePrefix #

timePrefix TODO

timePrefix

is optional
type: any of the folllowing: string or array (timePrefix)
cannot be null
defined in: timePrefix

bucket #

Top level S3 bucket under which to find data, metadata, or namespace folders

bucket

is optional
type: string
cannot be null
Unstructured type

namespace #

Namespace separation of data files. Folder path under top level S3 bucket in which to find related data files

namespace

is optional
type: string
cannot be null
Unstructured type

s3ListObjectsAll #

s3ListObjectsAll TODO

s3ListObjectsAll

is optional
type: boolean
cannot be null
Unstructured type

objectTimeRange #

objectTimeRange TODO

objectTimeRange

is optional
type: string
cannot be null
Unstructured type

s3Sql #

S3 SQL statement to select rows of data to be returned from the files in the dataset

s3Sql

is optional
type: string
cannot be null
Unstructured type

suffixFilter #

Only process objects whose key ends with the suffixFilter

suffixFilter

is optional
type: string
cannot be null
Unstructured type

maxObjects #

maxObjects TODO

maxObjects

is optional
type: number
cannot be null
Unstructured type

maxObjectsDirection #

Enumeration of strings with constraints below

maxObjectsDirection

is optional
type: unknown (maxObjectsDirection)
cannot be null
defined in: maxObjectsDirection

maxObjectsDirection Constraints #

enum: the value of this property must be equal to one of the following values:

Value	Explanation
`"left"`
`"right"`

style #

Enumeration of strings with constraints below

style

is optional
type: unknown (style)
cannot be null
defined in: style

style Constraints #

enum: the value of this property must be equal to one of the following values:

Value	Explanation
`"conventional"`	Line delimitted JSON, gzipped. Filename ending: .jsonl.gz
`"kinesis-firehose"`	Line delimitted JSON, uncompressed text. Filename ending: .gz

extractTime #

JSONata expression to extract time from S3 object key, used by objectTimeRange filtering

extractTime

is optional
type: string
cannot be null
Unstructured type