2020/12/15 - AWS IoT Analytics - 5 updated api methods
Changes FileFormatConfiguration enables data store to save data in JSON or Parquet format. S3Paths enables you to specify the S3 objects that save your channel messages when you reprocess the pipeline.
{'fileFormatConfiguration': {'jsonConfiguration': {},
'parquetConfiguration': {'schemaDefinition': {'columns': [{'name': 'string',
'type': 'string'}]}}}}
Creates a data store, which is a repository for messages.
See also: AWS API Documentation
Request Syntax
client.create_datastore(
datastoreName='string',
datastoreStorage={
'serviceManagedS3': {}
,
'customerManagedS3': {
'bucket': 'string',
'keyPrefix': 'string',
'roleArn': 'string'
}
},
retentionPeriod={
'unlimited': True|False,
'numberOfDays': 123
},
tags=[
{
'key': 'string',
'value': 'string'
},
],
fileFormatConfiguration={
'jsonConfiguration': {}
,
'parquetConfiguration': {
'schemaDefinition': {
'columns': [
{
'name': 'string',
'type': 'string'
},
]
}
}
}
)
string
[REQUIRED]
The name of the data store.
dict
Where data store data is stored. You can choose one of serviceManagedS3 or customerManagedS3 storage. If not specified, the default is serviceManagedS3 . You cannot change this storage option after the data store is created.
serviceManagedS3 (dict) --
Use this to store data store data in an S3 bucket managed by AWS IoT Analytics. You cannot change the choice of service-managed or customer-managed S3 storage after the data store is created.
customerManagedS3 (dict) --
Use this to store data store data in an S3 bucket that you manage. When customer managed storage is selected, the retentionPeriod parameter is ignored. The choice of service-managed or customer-managed S3 storage cannot be changed after creation of the data store.
bucket (string) -- [REQUIRED]
The name of the S3 bucket in which data store data is stored.
keyPrefix (string) --
Optional. The prefix used to create the keys of the data store data objects. Each object in an S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
roleArn (string) -- [REQUIRED]
The ARN of the role that grants AWS IoT Analytics permission to interact with your Amazon S3 resources.
dict
How long, in days, message data is kept for the data store. When customerManagedS3 storage is selected, this parameter is ignored.
unlimited (boolean) --
If true, message data is kept indefinitely.
numberOfDays (integer) --
The number of days that message data is kept. The unlimited parameter must be false.
list
Metadata which can be used to manage the data store.
(dict) --
A set of key-value pairs that are used to manage the resource.
key (string) -- [REQUIRED]
The tag's key.
value (string) -- [REQUIRED]
The tag's value.
dict
Contains the configuration information of file formats. AWS IoT Analytics data stores support JSON and Parquet .
The default file format is JSON. You can specify only one format.
You can't change the file format after you create the data store.
jsonConfiguration (dict) --
Contains the configuration information of the JSON format.
parquetConfiguration (dict) --
Contains the configuration information of the Parquet format.
schemaDefinition (dict) --
Information needed to define a schema.
columns (list) --
Specifies one or more columns that store your data.
Each schema can have up to 100 columns. Each column can have up to 100 nested types
(dict) --
Contains information about a column that stores your data.
name (string) -- [REQUIRED]
The name of the column.
type (string) -- [REQUIRED]
The type of data. For more information about the supported data types, see Common data types in the AWS Glue Developer Guide .
dict
Response Syntax
{
'datastoreName': 'string',
'datastoreArn': 'string',
'retentionPeriod': {
'unlimited': True|False,
'numberOfDays': 123
}
}
Response Structure
(dict) --
datastoreName (string) --
The name of the data store.
datastoreArn (string) --
The ARN of the data store.
retentionPeriod (dict) --
How long, in days, message data is kept for the data store.
unlimited (boolean) --
If true, message data is kept indefinitely.
numberOfDays (integer) --
The number of days that message data is kept. The unlimited parameter must be false.
{'datastore': {'fileFormatConfiguration': {'jsonConfiguration': {},
'parquetConfiguration': {'schemaDefinition': {'columns': [{'name': 'string',
'type': 'string'}]}}}}}
Retrieves information about a data store.
See also: AWS API Documentation
Request Syntax
client.describe_datastore(
datastoreName='string',
includeStatistics=True|False
)
string
[REQUIRED]
The name of the data store
boolean
If true, additional statistical information about the data store is included in the response. This feature cannot be used with a data store whose S3 storage is customer-managed.
dict
Response Syntax
{
'datastore': {
'name': 'string',
'storage': {
'serviceManagedS3': {},
'customerManagedS3': {
'bucket': 'string',
'keyPrefix': 'string',
'roleArn': 'string'
}
},
'arn': 'string',
'status': 'CREATING'|'ACTIVE'|'DELETING',
'retentionPeriod': {
'unlimited': True|False,
'numberOfDays': 123
},
'creationTime': datetime(2015, 1, 1),
'lastUpdateTime': datetime(2015, 1, 1),
'lastMessageArrivalTime': datetime(2015, 1, 1),
'fileFormatConfiguration': {
'jsonConfiguration': {},
'parquetConfiguration': {
'schemaDefinition': {
'columns': [
{
'name': 'string',
'type': 'string'
},
]
}
}
}
},
'statistics': {
'size': {
'estimatedSizeInBytes': 123.0,
'estimatedOn': datetime(2015, 1, 1)
}
}
}
Response Structure
(dict) --
datastore (dict) --
Information about the data store.
name (string) --
The name of the data store.
storage (dict) --
Where data store data is stored. You can choose one of serviceManagedS3 or customerManagedS3 storage. If not specified, the default is serviceManagedS3 . You cannot change this storage option after the data store is created.
serviceManagedS3 (dict) --
Use this to store data store data in an S3 bucket managed by AWS IoT Analytics. You cannot change the choice of service-managed or customer-managed S3 storage after the data store is created.
customerManagedS3 (dict) --
Use this to store data store data in an S3 bucket that you manage. When customer managed storage is selected, the retentionPeriod parameter is ignored. The choice of service-managed or customer-managed S3 storage cannot be changed after creation of the data store.
bucket (string) --
The name of the S3 bucket in which data store data is stored.
keyPrefix (string) --
Optional. The prefix used to create the keys of the data store data objects. Each object in an S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
roleArn (string) --
The ARN of the role that grants AWS IoT Analytics permission to interact with your Amazon S3 resources.
arn (string) --
The ARN of the data store.
status (string) --
The status of a data store:
CREATING
The data store is being created.
ACTIVE
The data store has been created and can be used.
DELETING
The data store is being deleted.
retentionPeriod (dict) --
How long, in days, message data is kept for the data store. When customerManagedS3 storage is selected, this parameter is ignored.
unlimited (boolean) --
If true, message data is kept indefinitely.
numberOfDays (integer) --
The number of days that message data is kept. The unlimited parameter must be false.
creationTime (datetime) --
When the data store was created.
lastUpdateTime (datetime) --
The last time the data store was updated.
lastMessageArrivalTime (datetime) --
The last time when a new message arrived in the data store.
AWS IoT Analytics updates this value at most once per minute for one data store. Hence, the lastMessageArrivalTime value is an approximation.
This feature only applies to messages that arrived in the data store after October 23, 2020.
fileFormatConfiguration (dict) --
Contains the configuration information of file formats. AWS IoT Analytics data stores support JSON and Parquet .
The default file format is JSON. You can specify only one format.
You can't change the file format after you create the data store.
jsonConfiguration (dict) --
Contains the configuration information of the JSON format.
parquetConfiguration (dict) --
Contains the configuration information of the Parquet format.
schemaDefinition (dict) --
Information needed to define a schema.
columns (list) --
Specifies one or more columns that store your data.
Each schema can have up to 100 columns. Each column can have up to 100 nested types
(dict) --
Contains information about a column that stores your data.
name (string) --
The name of the column.
type (string) --
The type of data. For more information about the supported data types, see Common data types in the AWS Glue Developer Guide .
statistics (dict) --
Additional statistical information about the data store. Included if the includeStatistics parameter is set to true in the request.
size (dict) --
The estimated size of the data store.
estimatedSizeInBytes (float) --
The estimated size of the resource, in bytes.
estimatedOn (datetime) --
The time when the estimate of the size of the resource was made.
{'datastoreSummaries': {'fileFormatType': 'JSON | PARQUET'}}
Retrieves a list of data stores.
See also: AWS API Documentation
Request Syntax
client.list_datastores(
nextToken='string',
maxResults=123
)
string
The token for the next set of results.
integer
The maximum number of results to return in this request.
The default value is 100.
dict
Response Syntax
{
'datastoreSummaries': [
{
'datastoreName': 'string',
'datastoreStorage': {
'serviceManagedS3': {},
'customerManagedS3': {
'bucket': 'string',
'keyPrefix': 'string',
'roleArn': 'string'
}
},
'status': 'CREATING'|'ACTIVE'|'DELETING',
'creationTime': datetime(2015, 1, 1),
'lastUpdateTime': datetime(2015, 1, 1),
'lastMessageArrivalTime': datetime(2015, 1, 1),
'fileFormatType': 'JSON'|'PARQUET'
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
datastoreSummaries (list) --
A list of DatastoreSummary objects.
(dict) --
A summary of information about a data store.
datastoreName (string) --
The name of the data store.
datastoreStorage (dict) --
Where data store data is stored.
serviceManagedS3 (dict) --
Used to store data store data in an S3 bucket managed by AWS IoT Analytics.
customerManagedS3 (dict) --
Used to store data store data in an S3 bucket that you manage.
bucket (string) --
The name of the S3 bucket in which data store data is stored.
keyPrefix (string) --
Optional. The prefix used to create the keys of the data store data objects. Each object in an S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
roleArn (string) --
The ARN of the role that grants AWS IoT Analytics permission to interact with your Amazon S3 resources.
status (string) --
The status of the data store.
creationTime (datetime) --
When the data store was created.
lastUpdateTime (datetime) --
The last time the data store was updated.
lastMessageArrivalTime (datetime) --
The last time when a new message arrived in the data store.
AWS IoT Analytics updates this value at most once per minute for one data store. Hence, the lastMessageArrivalTime value is an approximation.
This feature only applies to messages that arrived in the data store after October 23, 2020.
fileFormatType (string) --
The file format of the data in the data store.
nextToken (string) --
The token to retrieve the next set of results, or null if there are no more results.
{'channelMessages': {'s3Paths': ['string']}}
Starts the reprocessing of raw message data through the pipeline.
See also: AWS API Documentation
Request Syntax
client.start_pipeline_reprocessing(
pipelineName='string',
startTime=datetime(2015, 1, 1),
endTime=datetime(2015, 1, 1),
channelMessages={
's3Paths': [
'string',
]
}
)
string
[REQUIRED]
The name of the pipeline on which to start reprocessing.
datetime
The start time (inclusive) of raw message data that is reprocessed.
If you specify a value for the startTime parameter, you must not use the channelMessages object.
datetime
The end time (exclusive) of raw message data that is reprocessed.
If you specify a value for the endTime parameter, you must not use the channelMessages object.
dict
Specifies one or more sets of channel messages that you want to reprocess.
If you use the channelMessages object, you must not specify a value for startTime and endTime .
s3Paths (list) --
Specifies one or more keys that identify the Amazon Simple Storage Service (Amazon S3) objects that save your channel messages.
(string) --
dict
Response Syntax
{
'reprocessingId': 'string'
}
Response Structure
(dict) --
reprocessingId (string) --
The ID of the pipeline reprocessing activity that was started.
{'fileFormatConfiguration': {'jsonConfiguration': {},
'parquetConfiguration': {'schemaDefinition': {'columns': [{'name': 'string',
'type': 'string'}]}}}}
Updates the settings of a data store.
See also: AWS API Documentation
Request Syntax
client.update_datastore(
datastoreName='string',
retentionPeriod={
'unlimited': True|False,
'numberOfDays': 123
},
datastoreStorage={
'serviceManagedS3': {}
,
'customerManagedS3': {
'bucket': 'string',
'keyPrefix': 'string',
'roleArn': 'string'
}
},
fileFormatConfiguration={
'jsonConfiguration': {}
,
'parquetConfiguration': {
'schemaDefinition': {
'columns': [
{
'name': 'string',
'type': 'string'
},
]
}
}
}
)
string
[REQUIRED]
The name of the data store to be updated.
dict
How long, in days, message data is kept for the data store. The retention period cannot be updated if the data store's S3 storage is customer-managed.
unlimited (boolean) --
If true, message data is kept indefinitely.
numberOfDays (integer) --
The number of days that message data is kept. The unlimited parameter must be false.
dict
Where data store data is stored. You can choose one of serviceManagedS3 or customerManagedS3 storage. If not specified, the default is``serviceManagedS3`` . You cannot change this storage option after the data store is created.
serviceManagedS3 (dict) --
Use this to store data store data in an S3 bucket managed by AWS IoT Analytics. You cannot change the choice of service-managed or customer-managed S3 storage after the data store is created.
customerManagedS3 (dict) --
Use this to store data store data in an S3 bucket that you manage. When customer managed storage is selected, the retentionPeriod parameter is ignored. The choice of service-managed or customer-managed S3 storage cannot be changed after creation of the data store.
bucket (string) -- [REQUIRED]
The name of the S3 bucket in which data store data is stored.
keyPrefix (string) --
Optional. The prefix used to create the keys of the data store data objects. Each object in an S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
roleArn (string) -- [REQUIRED]
The ARN of the role that grants AWS IoT Analytics permission to interact with your Amazon S3 resources.
dict
Contains the configuration information of file formats. AWS IoT Analytics data stores support JSON and Parquet .
The default file format is JSON. You can specify only one format.
You can't change the file format after you create the data store.
jsonConfiguration (dict) --
Contains the configuration information of the JSON format.
parquetConfiguration (dict) --
Contains the configuration information of the Parquet format.
schemaDefinition (dict) --
Information needed to define a schema.
columns (list) --
Specifies one or more columns that store your data.
Each schema can have up to 100 columns. Each column can have up to 100 nested types
(dict) --
Contains information about a column that stores your data.
name (string) -- [REQUIRED]
The name of the column.
type (string) -- [REQUIRED]
The type of data. For more information about the supported data types, see Common data types in the AWS Glue Developer Guide .
None