2018/08/30 - Amazon SageMaker Service - 4 updated api methods
Changes VolumeKmsKeyId now available in Batch Transform Job
{'TransformResources': {'VolumeKmsKeyId': 'string'}}
Starts a transform job. A transform job uses a trained model to get inferences on a dataset and saves these results to an Amazon S3 location that you specify.
To perform batch transformations, you create a transform job and use the data that you have readily available.
In the request body, you provide the following:
TransformJobName - Identifies the transform job. The name must be unique within an AWS Region in an AWS account.
ModelName - Identifies the model to use. ModelName must be the name of an existing Amazon SageMaker model in the same AWS Region and AWS account. For information on creating a model, see CreateModel .
TransformInput - Describes the dataset to be transformed and the Amazon S3 location where it is stored.
TransformOutput - Identifies the Amazon S3 location where you want Amazon SageMaker to save the results from the transform job.
TransformResources - Identifies the ML compute instances for the transform job.
For more information about how batch transformation works Amazon SageMaker, see How It Works .
See also: AWS API Documentation
Request Syntax
client.create_transform_job(
TransformJobName='string',
ModelName='string',
MaxConcurrentTransforms=123,
MaxPayloadInMB=123,
BatchStrategy='MultiRecord'|'SingleRecord',
Environment={
'string': 'string'
},
TransformInput={
'DataSource': {
'S3DataSource': {
'S3DataType': 'ManifestFile'|'S3Prefix',
'S3Uri': 'string'
}
},
'ContentType': 'string',
'CompressionType': 'None'|'Gzip',
'SplitType': 'None'|'Line'|'RecordIO'
},
TransformOutput={
'S3OutputPath': 'string',
'Accept': 'string',
'AssembleWith': 'None'|'Line',
'KmsKeyId': 'string'
},
TransformResources={
'InstanceType': 'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge',
'InstanceCount': 123,
'VolumeKmsKeyId': 'string'
},
Tags=[
{
'Key': 'string',
'Value': 'string'
},
]
)
string
[REQUIRED]
The name of the transform job. The name must be unique within an AWS Region in an AWS account.
string
[REQUIRED]
The name of the model that you want to use for the transform job. ModelName must be the name of an existing Amazon SageMaker model within an AWS Region in an AWS account.
integer
The maximum number of parallel requests that can be sent to each instance in a transform job. This is good for algorithms that implement multiple workers on larger instances . The default value is 1 . To allow Amazon SageMaker to determine the appropriate number for MaxConcurrentTransforms , set the value to 0 .
integer
The maximum payload size allowed, in MB. A payload is the data portion of a record (without metadata). The value in MaxPayloadInMB must be greater or equal to the size of a single record. You can approximate the size of a record by dividing the size of your dataset by the number of records. Then multiply this value by the number of records you want in a mini-batch. It is recommended to enter a value slightly larger than this to ensure the records fit within the maximum payload size. The default value is 6 MB. For an unlimited payload size, set the value to 0 .
string
Determines the number of records included in a single mini-batch. SingleRecord means only one record is used per mini-batch. MultiRecord means a mini-batch is set to contain as many records that can fit within the MaxPayloadInMB limit.
Batch transform will automatically split your input data into whatever payload size is specified if you set SplitType to Line and BatchStrategy to MultiRecord . There's no need to split the dataset into smaller files or to use larger payload sizes unless the records in your dataset are very large.
dict
The environment variables to set in the Docker container. We support up to 16 key and values entries in the map.
(string) --
(string) --
dict
[REQUIRED]
Describes the input source and the way the transform job consumes it.
DataSource (dict) -- [REQUIRED]
Describes the location of the channel data, meaning the S3 location of the input data that the model can consume.
S3DataSource (dict) -- [REQUIRED]
The S3 location of the data source that is associated with a channel.
S3DataType (string) -- [REQUIRED]
If you choose S3Prefix , S3Uri identifies a key name prefix. Amazon SageMaker uses all objects with the specified key name prefix for batch transform.
If you choose ManifestFile , S3Uri identifies an object that is a manifest file containing a list of object keys that you want Amazon SageMaker to use for batch transform.
S3Uri (string) -- [REQUIRED]
Depending on the value specified for the S3DataType , identifies either a key name prefix or a manifest. For example:
A key name prefix might look like this: s3://bucketname/exampleprefix .
A manifest might look like this: s3://bucketname/example.manifest The manifest is an S3 object which is a JSON file with the following format: [ {"prefix": "s3://customer_bucket/some/prefix/"}, "relative/path/to/custdata-1", "relative/path/custdata-2", ... ] The preceding JSON matches the following S3Uris : s3://customer_bucket/some/prefix/relative/path/to/custdata-1 s3://customer_bucket/some/prefix/relative/path/custdata-1 ... The complete set of S3Uris in this manifest constitutes the input data for the channel for this datasource. The object that each S3Uris points to must be readable by the IAM role that Amazon SageMaker uses to perform tasks on your behalf.
ContentType (string) --
The multipurpose internet mail extension (MIME) type of the data. Amazon SageMaker uses the MIME type with each http call to transfer data to the transform job.
CompressionType (string) --
Compressing data helps save on storage space. If your transform data is compressed, specify the compression type.and Amazon SageMaker will automatically decompress the data for the transform job accordingly. The default value is None .
SplitType (string) --
The method to use to split the transform job's data into smaller batches. The default value is None . If you don't want to split the data, specify None . If you want to split records on a newline character boundary, specify Line . To split records according to the RecordIO format, specify RecordIO .
Amazon SageMaker will send maximum number of records per batch in each request up to the MaxPayloadInMB limit. For more information, see RecordIO data format .
Note
For information about the RecordIO format, see Data Format .
dict
[REQUIRED]
Describes the results of the transform job.
S3OutputPath (string) -- [REQUIRED]
The Amazon S3 path where you want Amazon SageMaker to store the results of the transform job. For example, s3://bucket-name/key-name-prefix .
For every S3 object used as input for the transform job, the transformed data is stored in a corresponding subfolder in the location under the output prefix. For example, the input data s3://bucket-name/input-name-prefix/dataset01/data.csv will have the transformed data stored at s3://bucket-name/key-name-prefix/dataset01/ , based on the original name, as a series of .part files (.part0001, part0002, etc).
Accept (string) --
The MIME type used to specify the output data. Amazon SageMaker uses the MIME type with each http call to transfer data from the transform job.
AssembleWith (string) --
Defines how to assemble the results of the transform job as a single S3 object. You should select a format that is most convenient to you. To concatenate the results in binary format, specify None . To add a newline character at the end of every transformed record, specify Line . To assemble the output in RecordIO format, specify RecordIO . The default value is None .
For information about the RecordIO format, see Data Format .
KmsKeyId (string) --
The AWS Key Management Service (AWS KMS) key for Amazon S3 server-side encryption that Amazon SageMaker uses to encrypt the transformed data.
If you don't provide a KMS key ID, Amazon SageMaker uses the default KMS key for Amazon S3 for your role's account. For more information, see KMS-Managed Encryption Keys in the Amazon Simple Storage Service Developer Guide.
The KMS key policy must grant permission to the IAM role that you specify in your CreateTramsformJob request. For more information, see Using Key Policies in AWS KMS in the AWS Key Management Service Developer Guide .
dict
[REQUIRED]
Describes the resources, including ML instance types and ML instance count, to use for the transform job.
InstanceType (string) -- [REQUIRED]
The ML compute instance type for the transform job. For using built-in algorithms to transform moderately sized datasets, ml.m4.xlarge or ml.m5.large should suffice. There is no default value for InstanceType .
InstanceCount (integer) -- [REQUIRED]
The number of ML compute instances to use in the transform job. For distributed transform, provide a value greater than 1. The default value is 1 .
VolumeKmsKeyId (string) --
The Amazon Resource Name (ARN) of a AWS Key Management Service key that Amazon SageMaker uses to encrypt data on the storage volume attached to the ML compute instance(s) that run the batch transform job.
list
An array of key-value pairs. Adding tags is optional. For more information, see Using Cost Allocation Tags in the AWS Billing and Cost Management User Guide .
(dict) --
Describes a tag.
Key (string) -- [REQUIRED]
The tag key.
Value (string) -- [REQUIRED]
The tag value.
dict
Response Syntax
{
'TransformJobArn': 'string'
}
Response Structure
(dict) --
TransformJobArn (string) --
The Amazon Resource Name (ARN) of the transform job.
{'EndpointStatus': ['SystemUpdating']}
Returns the description of an endpoint.
See also: AWS API Documentation
Request Syntax
client.describe_endpoint(
EndpointName='string'
)
string
[REQUIRED]
The name of the endpoint.
dict
Response Syntax
{
'EndpointName': 'string',
'EndpointArn': 'string',
'EndpointConfigName': 'string',
'ProductionVariants': [
{
'VariantName': 'string',
'DeployedImages': [
{
'SpecifiedImage': 'string',
'ResolvedImage': 'string',
'ResolutionTime': datetime(2015, 1, 1)
},
],
'CurrentWeight': ...,
'DesiredWeight': ...,
'CurrentInstanceCount': 123,
'DesiredInstanceCount': 123
},
],
'EndpointStatus': 'OutOfService'|'Creating'|'Updating'|'SystemUpdating'|'RollingBack'|'InService'|'Deleting'|'Failed',
'FailureReason': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastModifiedTime': datetime(2015, 1, 1)
}
Response Structure
(dict) --
EndpointName (string) --
Name of the endpoint.
EndpointArn (string) --
The Amazon Resource Name (ARN) of the endpoint.
EndpointConfigName (string) --
The name of the endpoint configuration associated with this endpoint.
ProductionVariants (list) --
An array of ProductionVariantSummary objects, one for each model hosted behind this endpoint.
(dict) --
Describes weight and capacities for a production variant associated with an endpoint. If you sent a request to the UpdateEndpointWeightsAndCapacities API and the endpoint status is Updating , you get different desired and current values.
VariantName (string) --
The name of the variant.
DeployedImages (list) --
An array of DeployedImage objects that specify the Amazon EC2 Container Registry paths of the inference images deployed on instances of this ProductionVariant .
(dict) --
Gets the Amazon EC2 Container Registry path of the docker image of the model that is hosted in this ProductionVariant .
If you used the registry/repository[:tag] form to to specify the image path of the primary container when you created the model hosted in this ProductionVariant , the path resolves to a path of the form registry/repository[@digest] . A digest is a hash value that identifies a specific version of an image. For information about Amazon ECR paths, see Pulling an Image in the Amazon ECR User Guide .
SpecifiedImage (string) --
The image path you specified when you created the model.
ResolvedImage (string) --
The specific digest path of the image hosted in this ProductionVariant .
ResolutionTime (datetime) --
The date and time when the image path for the model resolved to the ResolvedImage
CurrentWeight (float) --
The weight associated with the variant.
DesiredWeight (float) --
The requested weight, as specified in the UpdateEndpointWeightsAndCapacities request.
CurrentInstanceCount (integer) --
The number of instances associated with the variant.
DesiredInstanceCount (integer) --
The number of instances requested in the UpdateEndpointWeightsAndCapacities request.
EndpointStatus (string) --
The status of the endpoint.
FailureReason (string) --
If the status of the endpoint is Failed , the reason why it failed.
CreationTime (datetime) --
A timestamp that shows when the endpoint was created.
LastModifiedTime (datetime) --
A timestamp that shows when the endpoint was last modified.
{'TransformResources': {'VolumeKmsKeyId': 'string'}}
Returns information about a transform job.
See also: AWS API Documentation
Request Syntax
client.describe_transform_job(
TransformJobName='string'
)
string
[REQUIRED]
The name of the transform job that you want to view details of.
dict
Response Syntax
{
'TransformJobName': 'string',
'TransformJobArn': 'string',
'TransformJobStatus': 'InProgress'|'Completed'|'Failed'|'Stopping'|'Stopped',
'FailureReason': 'string',
'ModelName': 'string',
'MaxConcurrentTransforms': 123,
'MaxPayloadInMB': 123,
'BatchStrategy': 'MultiRecord'|'SingleRecord',
'Environment': {
'string': 'string'
},
'TransformInput': {
'DataSource': {
'S3DataSource': {
'S3DataType': 'ManifestFile'|'S3Prefix',
'S3Uri': 'string'
}
},
'ContentType': 'string',
'CompressionType': 'None'|'Gzip',
'SplitType': 'None'|'Line'|'RecordIO'
},
'TransformOutput': {
'S3OutputPath': 'string',
'Accept': 'string',
'AssembleWith': 'None'|'Line',
'KmsKeyId': 'string'
},
'TransformResources': {
'InstanceType': 'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge',
'InstanceCount': 123,
'VolumeKmsKeyId': 'string'
},
'CreationTime': datetime(2015, 1, 1),
'TransformStartTime': datetime(2015, 1, 1),
'TransformEndTime': datetime(2015, 1, 1)
}
Response Structure
(dict) --
TransformJobName (string) --
The name of the transform job.
TransformJobArn (string) --
The Amazon Resource Name (ARN) of the transform job.
TransformJobStatus (string) --
The status of the transform job. If the transform job failed, the reason is returned in the FailureReason field.
FailureReason (string) --
If the transform job failed, the reason that it failed.
ModelName (string) --
The name of the model used in the transform job.
MaxConcurrentTransforms (integer) --
The maximum number of parallel requests on each instance node that can be launched in a transform job. The default value is 1.
MaxPayloadInMB (integer) --
The maximum payload size , in MB used in the transform job.
BatchStrategy (string) --
SingleRecord means only one record was used per a batch. MultiRecord means batches contained as many records that could possibly fit within the MaxPayloadInMB limit.
Environment (dict) --
(string) --
(string) --
TransformInput (dict) --
Describes the dataset to be transformed and the Amazon S3 location where it is stored.
DataSource (dict) --
Describes the location of the channel data, meaning the S3 location of the input data that the model can consume.
S3DataSource (dict) --
The S3 location of the data source that is associated with a channel.
S3DataType (string) --
If you choose S3Prefix , S3Uri identifies a key name prefix. Amazon SageMaker uses all objects with the specified key name prefix for batch transform.
If you choose ManifestFile , S3Uri identifies an object that is a manifest file containing a list of object keys that you want Amazon SageMaker to use for batch transform.
S3Uri (string) --
Depending on the value specified for the S3DataType , identifies either a key name prefix or a manifest. For example:
A key name prefix might look like this: s3://bucketname/exampleprefix .
A manifest might look like this: s3://bucketname/example.manifest The manifest is an S3 object which is a JSON file with the following format: [ {"prefix": "s3://customer_bucket/some/prefix/"}, "relative/path/to/custdata-1", "relative/path/custdata-2", ... ] The preceding JSON matches the following S3Uris : s3://customer_bucket/some/prefix/relative/path/to/custdata-1 s3://customer_bucket/some/prefix/relative/path/custdata-1 ... The complete set of S3Uris in this manifest constitutes the input data for the channel for this datasource. The object that each S3Uris points to must be readable by the IAM role that Amazon SageMaker uses to perform tasks on your behalf.
ContentType (string) --
The multipurpose internet mail extension (MIME) type of the data. Amazon SageMaker uses the MIME type with each http call to transfer data to the transform job.
CompressionType (string) --
Compressing data helps save on storage space. If your transform data is compressed, specify the compression type.and Amazon SageMaker will automatically decompress the data for the transform job accordingly. The default value is None .
SplitType (string) --
The method to use to split the transform job's data into smaller batches. The default value is None . If you don't want to split the data, specify None . If you want to split records on a newline character boundary, specify Line . To split records according to the RecordIO format, specify RecordIO .
Amazon SageMaker will send maximum number of records per batch in each request up to the MaxPayloadInMB limit. For more information, see RecordIO data format .
Note
For information about the RecordIO format, see Data Format .
TransformOutput (dict) --
Identifies the Amazon S3 location where you want Amazon SageMaker to save the results from the transform job.
S3OutputPath (string) --
The Amazon S3 path where you want Amazon SageMaker to store the results of the transform job. For example, s3://bucket-name/key-name-prefix .
For every S3 object used as input for the transform job, the transformed data is stored in a corresponding subfolder in the location under the output prefix. For example, the input data s3://bucket-name/input-name-prefix/dataset01/data.csv will have the transformed data stored at s3://bucket-name/key-name-prefix/dataset01/ , based on the original name, as a series of .part files (.part0001, part0002, etc).
Accept (string) --
The MIME type used to specify the output data. Amazon SageMaker uses the MIME type with each http call to transfer data from the transform job.
AssembleWith (string) --
Defines how to assemble the results of the transform job as a single S3 object. You should select a format that is most convenient to you. To concatenate the results in binary format, specify None . To add a newline character at the end of every transformed record, specify Line . To assemble the output in RecordIO format, specify RecordIO . The default value is None .
For information about the RecordIO format, see Data Format .
KmsKeyId (string) --
The AWS Key Management Service (AWS KMS) key for Amazon S3 server-side encryption that Amazon SageMaker uses to encrypt the transformed data.
If you don't provide a KMS key ID, Amazon SageMaker uses the default KMS key for Amazon S3 for your role's account. For more information, see KMS-Managed Encryption Keys in the Amazon Simple Storage Service Developer Guide.
The KMS key policy must grant permission to the IAM role that you specify in your CreateTramsformJob request. For more information, see Using Key Policies in AWS KMS in the AWS Key Management Service Developer Guide .
TransformResources (dict) --
Describes the resources, including ML instance types and ML instance count, to use for the transform job.
InstanceType (string) --
The ML compute instance type for the transform job. For using built-in algorithms to transform moderately sized datasets, ml.m4.xlarge or ml.m5.large should suffice. There is no default value for InstanceType .
InstanceCount (integer) --
The number of ML compute instances to use in the transform job. For distributed transform, provide a value greater than 1. The default value is 1 .
VolumeKmsKeyId (string) --
The Amazon Resource Name (ARN) of a AWS Key Management Service key that Amazon SageMaker uses to encrypt data on the storage volume attached to the ML compute instance(s) that run the batch transform job.
CreationTime (datetime) --
A timestamp that shows when the transform Job was created.
TransformStartTime (datetime) --
Indicates when the transform job starts on ML instances. You are billed for the time interval between this time and the value of TransformEndTime .
TransformEndTime (datetime) --
Indicates when the transform job is Completed , Stopped , or Failed . You are billed for the time interval between this time and the value of TransformStartTime .
{'StatusEquals': ['SystemUpdating']}
Response {'Endpoints': {'EndpointStatus': ['SystemUpdating']}}
Lists endpoints.
See also: AWS API Documentation
Request Syntax
client.list_endpoints(
SortBy='Name'|'CreationTime'|'Status',
SortOrder='Ascending'|'Descending',
NextToken='string',
MaxResults=123,
NameContains='string',
CreationTimeBefore=datetime(2015, 1, 1),
CreationTimeAfter=datetime(2015, 1, 1),
LastModifiedTimeBefore=datetime(2015, 1, 1),
LastModifiedTimeAfter=datetime(2015, 1, 1),
StatusEquals='OutOfService'|'Creating'|'Updating'|'SystemUpdating'|'RollingBack'|'InService'|'Deleting'|'Failed'
)
string
Sorts the list of results. The default is CreationTime .
string
The sort order for results. The default is Ascending .
string
If the result of a ListEndpoints request was truncated, the response includes a NextToken . To retrieve the next set of endpoints, use the token in the next request.
integer
The maximum number of endpoints to return in the response.
string
A string in endpoint names. This filter returns only endpoints whose name contains the specified string.
datetime
A filter that returns only endpoints that were created before the specified time (timestamp).
datetime
A filter that returns only endpoints that were created after the specified time (timestamp).
datetime
A filter that returns only endpoints that were modified before the specified timestamp.
datetime
A filter that returns only endpoints that were modified after the specified timestamp.
string
A filter that returns only endpoints with the specified status.
dict
Response Syntax
{
'Endpoints': [
{
'EndpointName': 'string',
'EndpointArn': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastModifiedTime': datetime(2015, 1, 1),
'EndpointStatus': 'OutOfService'|'Creating'|'Updating'|'SystemUpdating'|'RollingBack'|'InService'|'Deleting'|'Failed'
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Endpoints (list) --
An array or endpoint objects.
(dict) --
Provides summary information for an endpoint.
EndpointName (string) --
The name of the endpoint.
EndpointArn (string) --
The Amazon Resource Name (ARN) of the endpoint.
CreationTime (datetime) --
A timestamp that shows when the endpoint was created.
LastModifiedTime (datetime) --
A timestamp that shows when the endpoint was last modified.
EndpointStatus (string) --
The status of the endpoint.
NextToken (string) --
If the response is truncated, Amazon SageMaker returns this token. To retrieve the next set of training jobs, use it in the subsequent request.