2021/07/27 - AWS IoT Analytics - 4 updated api methods
Changes IoT Analytics now supports creating a dataset resource with IoT SiteWise MultiLayerStorage data stores, enabling customers to query industrial data within the service. This release includes adding JOIN functionality for customers to query multiple data sources in a dataset.
{'datastoreStorage': {'iotSiteWiseMultiLayerStorage': {'customerManagedS3Storage': {'bucket': 'string',
'keyPrefix': 'string'}}}}
Creates a data store, which is a repository for messages.
See also: AWS API Documentation
Request Syntax
client.create_datastore(
datastoreName='string',
datastoreStorage={
'serviceManagedS3': {}
,
'customerManagedS3': {
'bucket': 'string',
'keyPrefix': 'string',
'roleArn': 'string'
},
'iotSiteWiseMultiLayerStorage': {
'customerManagedS3Storage': {
'bucket': 'string',
'keyPrefix': 'string'
}
}
},
retentionPeriod={
'unlimited': True|False,
'numberOfDays': 123
},
tags=[
{
'key': 'string',
'value': 'string'
},
],
fileFormatConfiguration={
'jsonConfiguration': {}
,
'parquetConfiguration': {
'schemaDefinition': {
'columns': [
{
'name': 'string',
'type': 'string'
},
]
}
}
},
datastorePartitions={
'partitions': [
{
'attributePartition': {
'attributeName': 'string'
},
'timestampPartition': {
'attributeName': 'string',
'timestampFormat': 'string'
}
},
]
}
)
string
[REQUIRED]
The name of the data store.
dict
Where data in a data store is stored.. You can choose serviceManagedS3 storage, customerManagedS3 storage, or iotSiteWiseMultiLayerStorage storage. The default is serviceManagedS3 . You can't change the choice of Amazon S3 storage after your data store is created.
serviceManagedS3 (dict) --
Used to store data in an Amazon S3 bucket managed by IoT Analytics. You can't change the choice of Amazon S3 storage after your data store is created.
customerManagedS3 (dict) --
S3-customer-managed; When you choose customer-managed storage, the retentionPeriod parameter is ignored. You can't change the choice of Amazon S3 storage after your data store is created.
bucket (string) -- [REQUIRED]
The name of the Amazon S3 bucket where your data is stored.
keyPrefix (string) --
(Optional) The prefix used to create the keys of the data store data objects. Each object in an Amazon S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
roleArn (string) -- [REQUIRED]
The ARN of the role that grants IoT Analytics permission to interact with your Amazon S3 resources.
iotSiteWiseMultiLayerStorage (dict) --
Used to store data used by IoT SiteWise in an Amazon S3 bucket that you manage. You can't change the choice of Amazon S3 storage after your data store is created.
customerManagedS3Storage (dict) -- [REQUIRED]
Used to store data used by IoT SiteWise in an Amazon S3 bucket that you manage.
bucket (string) -- [REQUIRED]
The name of the Amazon S3 bucket where your data is stored.
keyPrefix (string) --
(Optional) The prefix used to create the keys of the data store data objects. Each object in an Amazon S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
dict
How long, in days, message data is kept for the data store. When customerManagedS3 storage is selected, this parameter is ignored.
unlimited (boolean) --
If true, message data is kept indefinitely.
numberOfDays (integer) --
The number of days that message data is kept. The unlimited parameter must be false.
list
Metadata which can be used to manage the data store.
(dict) --
A set of key-value pairs that are used to manage the resource.
key (string) -- [REQUIRED]
The tag's key.
value (string) -- [REQUIRED]
The tag's value.
dict
Contains the configuration information of file formats. IoT Analytics data stores support JSON and Parquet .
The default file format is JSON. You can specify only one format.
You can't change the file format after you create the data store.
jsonConfiguration (dict) --
Contains the configuration information of the JSON format.
parquetConfiguration (dict) --
Contains the configuration information of the Parquet format.
schemaDefinition (dict) --
Information needed to define a schema.
columns (list) --
Specifies one or more columns that store your data.
Each schema can have up to 100 columns. Each column can have up to 100 nested types.
(dict) --
Contains information about a column that stores your data.
name (string) -- [REQUIRED]
The name of the column.
type (string) -- [REQUIRED]
The type of data. For more information about the supported data types, see Common data types in the Glue Developer Guide .
dict
Contains information about the partition dimensions in a data store.
partitions (list) --
A list of partition dimensions in a data store.
(dict) --
A single dimension to partition a data store. The dimension must be an AttributePartition or a TimestampPartition .
attributePartition (dict) --
A partition dimension defined by an attributeName .
attributeName (string) -- [REQUIRED]
The name of the attribute that defines a partition dimension.
timestampPartition (dict) --
A partition dimension defined by a timestamp attribute.
attributeName (string) -- [REQUIRED]
The attribute name of the partition defined by a timestamp.
timestampFormat (string) --
The timestamp format of a partition defined by a timestamp. The default format is seconds since epoch (January 1, 1970 at midnight UTC time).
dict
Response Syntax
{
'datastoreName': 'string',
'datastoreArn': 'string',
'retentionPeriod': {
'unlimited': True|False,
'numberOfDays': 123
}
}
Response Structure
(dict) --
datastoreName (string) --
The name of the data store.
datastoreArn (string) --
The ARN of the data store.
retentionPeriod (dict) --
How long, in days, message data is kept for the data store.
unlimited (boolean) --
If true, message data is kept indefinitely.
numberOfDays (integer) --
The number of days that message data is kept. The unlimited parameter must be false.
{'datastore': {'storage': {'iotSiteWiseMultiLayerStorage': {'customerManagedS3Storage': {'bucket': 'string',
'keyPrefix': 'string'}}}}}
Retrieves information about a data store.
See also: AWS API Documentation
Request Syntax
client.describe_datastore(
datastoreName='string',
includeStatistics=True|False
)
string
[REQUIRED]
The name of the data store
boolean
If true, additional statistical information about the data store is included in the response. This feature can't be used with a data store whose S3 storage is customer-managed.
dict
Response Syntax
{
'datastore': {
'name': 'string',
'storage': {
'serviceManagedS3': {},
'customerManagedS3': {
'bucket': 'string',
'keyPrefix': 'string',
'roleArn': 'string'
},
'iotSiteWiseMultiLayerStorage': {
'customerManagedS3Storage': {
'bucket': 'string',
'keyPrefix': 'string'
}
}
},
'arn': 'string',
'status': 'CREATING'|'ACTIVE'|'DELETING',
'retentionPeriod': {
'unlimited': True|False,
'numberOfDays': 123
},
'creationTime': datetime(2015, 1, 1),
'lastUpdateTime': datetime(2015, 1, 1),
'lastMessageArrivalTime': datetime(2015, 1, 1),
'fileFormatConfiguration': {
'jsonConfiguration': {},
'parquetConfiguration': {
'schemaDefinition': {
'columns': [
{
'name': 'string',
'type': 'string'
},
]
}
}
},
'datastorePartitions': {
'partitions': [
{
'attributePartition': {
'attributeName': 'string'
},
'timestampPartition': {
'attributeName': 'string',
'timestampFormat': 'string'
}
},
]
}
},
'statistics': {
'size': {
'estimatedSizeInBytes': 123.0,
'estimatedOn': datetime(2015, 1, 1)
}
}
}
Response Structure
(dict) --
datastore (dict) --
Information about the data store.
name (string) --
The name of the data store.
storage (dict) --
Where data in a data store is stored.. You can choose serviceManagedS3 storage, customerManagedS3 storage, or iotSiteWiseMultiLayerStorage storage. The default is serviceManagedS3 . You can't change the choice of Amazon S3 storage after your data store is created.
serviceManagedS3 (dict) --
Used to store data in an Amazon S3 bucket managed by IoT Analytics. You can't change the choice of Amazon S3 storage after your data store is created.
customerManagedS3 (dict) --
S3-customer-managed; When you choose customer-managed storage, the retentionPeriod parameter is ignored. You can't change the choice of Amazon S3 storage after your data store is created.
bucket (string) --
The name of the Amazon S3 bucket where your data is stored.
keyPrefix (string) --
(Optional) The prefix used to create the keys of the data store data objects. Each object in an Amazon S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
roleArn (string) --
The ARN of the role that grants IoT Analytics permission to interact with your Amazon S3 resources.
iotSiteWiseMultiLayerStorage (dict) --
Used to store data used by IoT SiteWise in an Amazon S3 bucket that you manage. You can't change the choice of Amazon S3 storage after your data store is created.
customerManagedS3Storage (dict) --
Used to store data used by IoT SiteWise in an Amazon S3 bucket that you manage.
bucket (string) --
The name of the Amazon S3 bucket where your data is stored.
keyPrefix (string) --
(Optional) The prefix used to create the keys of the data store data objects. Each object in an Amazon S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
arn (string) --
The ARN of the data store.
status (string) --
The status of a data store:
CREATING
The data store is being created.
ACTIVE
The data store has been created and can be used.
DELETING
The data store is being deleted.
retentionPeriod (dict) --
How long, in days, message data is kept for the data store. When customerManagedS3 storage is selected, this parameter is ignored.
unlimited (boolean) --
If true, message data is kept indefinitely.
numberOfDays (integer) --
The number of days that message data is kept. The unlimited parameter must be false.
creationTime (datetime) --
When the data store was created.
lastUpdateTime (datetime) --
The last time the data store was updated.
lastMessageArrivalTime (datetime) --
The last time when a new message arrived in the data store.
IoT Analytics updates this value at most once per minute for Amazon Simple Storage Service one data store. Hence, the lastMessageArrivalTime value is an approximation.
This feature only applies to messages that arrived in the data store after October 23, 2020.
fileFormatConfiguration (dict) --
Contains the configuration information of file formats. IoT Analytics data stores support JSON and Parquet .
The default file format is JSON. You can specify only one format.
You can't change the file format after you create the data store.
jsonConfiguration (dict) --
Contains the configuration information of the JSON format.
parquetConfiguration (dict) --
Contains the configuration information of the Parquet format.
schemaDefinition (dict) --
Information needed to define a schema.
columns (list) --
Specifies one or more columns that store your data.
Each schema can have up to 100 columns. Each column can have up to 100 nested types.
(dict) --
Contains information about a column that stores your data.
name (string) --
The name of the column.
type (string) --
The type of data. For more information about the supported data types, see Common data types in the Glue Developer Guide .
datastorePartitions (dict) --
Contains information about the partition dimensions in a data store.
partitions (list) --
A list of partition dimensions in a data store.
(dict) --
A single dimension to partition a data store. The dimension must be an AttributePartition or a TimestampPartition .
attributePartition (dict) --
A partition dimension defined by an attributeName .
attributeName (string) --
The name of the attribute that defines a partition dimension.
timestampPartition (dict) --
A partition dimension defined by a timestamp attribute.
attributeName (string) --
The attribute name of the partition defined by a timestamp.
timestampFormat (string) --
The timestamp format of a partition defined by a timestamp. The default format is seconds since epoch (January 1, 1970 at midnight UTC time).
statistics (dict) --
Additional statistical information about the data store. Included if the includeStatistics parameter is set to true in the request.
size (dict) --
The estimated size of the data store.
estimatedSizeInBytes (float) --
The estimated size of the resource, in bytes.
estimatedOn (datetime) --
The time when the estimate of the size of the resource was made.
{'datastoreSummaries': {'datastoreStorage': {'iotSiteWiseMultiLayerStorage': {'customerManagedS3Storage': {'bucket': 'string',
'keyPrefix': 'string'}}}}}
Retrieves a list of data stores.
See also: AWS API Documentation
Request Syntax
client.list_datastores(
nextToken='string',
maxResults=123
)
string
The token for the next set of results.
integer
The maximum number of results to return in this request.
The default value is 100.
dict
Response Syntax
{
'datastoreSummaries': [
{
'datastoreName': 'string',
'datastoreStorage': {
'serviceManagedS3': {},
'customerManagedS3': {
'bucket': 'string',
'keyPrefix': 'string',
'roleArn': 'string'
},
'iotSiteWiseMultiLayerStorage': {
'customerManagedS3Storage': {
'bucket': 'string',
'keyPrefix': 'string'
}
}
},
'status': 'CREATING'|'ACTIVE'|'DELETING',
'creationTime': datetime(2015, 1, 1),
'lastUpdateTime': datetime(2015, 1, 1),
'lastMessageArrivalTime': datetime(2015, 1, 1),
'fileFormatType': 'JSON'|'PARQUET',
'datastorePartitions': {
'partitions': [
{
'attributePartition': {
'attributeName': 'string'
},
'timestampPartition': {
'attributeName': 'string',
'timestampFormat': 'string'
}
},
]
}
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
datastoreSummaries (list) --
A list of DatastoreSummary objects.
(dict) --
A summary of information about a data store.
datastoreName (string) --
The name of the data store.
datastoreStorage (dict) --
Where data in a data store is stored.
serviceManagedS3 (dict) --
Used to store data in an Amazon S3 bucket managed by IoT Analytics.
customerManagedS3 (dict) --
Used to store data in an Amazon S3 bucket managed by IoT Analytics.
bucket (string) --
The name of the Amazon S3 bucket where your data is stored.
keyPrefix (string) --
(Optional) The prefix used to create the keys of the data store data objects. Each object in an Amazon S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
roleArn (string) --
The ARN of the role that grants IoT Analytics permission to interact with your Amazon S3 resources.
iotSiteWiseMultiLayerStorage (dict) --
Used to store data used by IoT SiteWise in an Amazon S3 bucket that you manage.
customerManagedS3Storage (dict) --
Used to store data used by IoT SiteWise in an Amazon S3 bucket that you manage.
bucket (string) --
The name of the Amazon S3 bucket where your data is stored.
keyPrefix (string) --
(Optional) The prefix used to create the keys of the data store data objects. Each object in an Amazon S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
status (string) --
The status of the data store.
creationTime (datetime) --
When the data store was created.
lastUpdateTime (datetime) --
The last time the data store was updated.
lastMessageArrivalTime (datetime) --
The last time when a new message arrived in the data store.
IoT Analytics updates this value at most once per minute for Amazon Simple Storage Service one data store. Hence, the lastMessageArrivalTime value is an approximation.
This feature only applies to messages that arrived in the data store after October 23, 2020.
fileFormatType (string) --
The file format of the data in the data store.
datastorePartitions (dict) --
Contains information about the partition dimensions in a data store.
partitions (list) --
A list of partition dimensions in a data store.
(dict) --
A single dimension to partition a data store. The dimension must be an AttributePartition or a TimestampPartition .
attributePartition (dict) --
A partition dimension defined by an attributeName .
attributeName (string) --
The name of the attribute that defines a partition dimension.
timestampPartition (dict) --
A partition dimension defined by a timestamp attribute.
attributeName (string) --
The attribute name of the partition defined by a timestamp.
timestampFormat (string) --
The timestamp format of a partition defined by a timestamp. The default format is seconds since epoch (January 1, 1970 at midnight UTC time).
nextToken (string) --
The token to retrieve the next set of results, or null if there are no more results.
{'datastoreStorage': {'iotSiteWiseMultiLayerStorage': {'customerManagedS3Storage': {'bucket': 'string',
'keyPrefix': 'string'}}}}
Used to update the settings of a data store.
See also: AWS API Documentation
Request Syntax
client.update_datastore(
datastoreName='string',
retentionPeriod={
'unlimited': True|False,
'numberOfDays': 123
},
datastoreStorage={
'serviceManagedS3': {}
,
'customerManagedS3': {
'bucket': 'string',
'keyPrefix': 'string',
'roleArn': 'string'
},
'iotSiteWiseMultiLayerStorage': {
'customerManagedS3Storage': {
'bucket': 'string',
'keyPrefix': 'string'
}
}
},
fileFormatConfiguration={
'jsonConfiguration': {}
,
'parquetConfiguration': {
'schemaDefinition': {
'columns': [
{
'name': 'string',
'type': 'string'
},
]
}
}
}
)
string
[REQUIRED]
The name of the data store to be updated.
dict
How long, in days, message data is kept for the data store. The retention period can't be updated if the data store's Amazon S3 storage is customer-managed.
unlimited (boolean) --
If true, message data is kept indefinitely.
numberOfDays (integer) --
The number of days that message data is kept. The unlimited parameter must be false.
dict
Where data in a data store is stored.. You can choose serviceManagedS3 storage, customerManagedS3 storage, or iotSiteWiseMultiLayerStorage storage. The default is serviceManagedS3 . You can't change the choice of Amazon S3 storage after your data store is created.
serviceManagedS3 (dict) --
Used to store data in an Amazon S3 bucket managed by IoT Analytics. You can't change the choice of Amazon S3 storage after your data store is created.
customerManagedS3 (dict) --
S3-customer-managed; When you choose customer-managed storage, the retentionPeriod parameter is ignored. You can't change the choice of Amazon S3 storage after your data store is created.
bucket (string) -- [REQUIRED]
The name of the Amazon S3 bucket where your data is stored.
keyPrefix (string) --
(Optional) The prefix used to create the keys of the data store data objects. Each object in an Amazon S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
roleArn (string) -- [REQUIRED]
The ARN of the role that grants IoT Analytics permission to interact with your Amazon S3 resources.
iotSiteWiseMultiLayerStorage (dict) --
Used to store data used by IoT SiteWise in an Amazon S3 bucket that you manage. You can't change the choice of Amazon S3 storage after your data store is created.
customerManagedS3Storage (dict) -- [REQUIRED]
Used to store data used by IoT SiteWise in an Amazon S3 bucket that you manage.
bucket (string) -- [REQUIRED]
The name of the Amazon S3 bucket where your data is stored.
keyPrefix (string) --
(Optional) The prefix used to create the keys of the data store data objects. Each object in an Amazon S3 bucket has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/).
dict
Contains the configuration information of file formats. IoT Analytics data stores support JSON and Parquet .
The default file format is JSON. You can specify only one format.
You can't change the file format after you create the data store.
jsonConfiguration (dict) --
Contains the configuration information of the JSON format.
parquetConfiguration (dict) --
Contains the configuration information of the Parquet format.
schemaDefinition (dict) --
Information needed to define a schema.
columns (list) --
Specifies one or more columns that store your data.
Each schema can have up to 100 columns. Each column can have up to 100 nested types.
(dict) --
Contains information about a column that stores your data.
name (string) -- [REQUIRED]
The name of the column.
type (string) -- [REQUIRED]
The type of data. For more information about the supported data types, see Common data types in the Glue Developer Guide .
None