2020/11/24 - AWS Batch - 2 updated api methods
Changes Add Ec2Configuration in ComputeEnvironment.ComputeResources. Use in CreateComputeEnvironment API to enable AmazonLinux2 support.
{'computeResources': {'ec2Configuration': [{'imageIdOverride': 'string',
'imageType': 'string'}]}}
Creates an AWS Batch compute environment. You can create MANAGED or UNMANAGED compute environments.
In a managed compute environment, AWS Batch manages the capacity and instance types of the compute resources within the environment. This is based on the compute resource specification that you define or the launch template that you specify when you create the compute environment. You can choose to use Amazon EC2 On-Demand Instances or Spot Instances in your managed compute environment. You can optionally set a maximum price so that Spot Instances only launch when the Spot Instance price is below a specified percentage of the On-Demand price.
Note
Multi-node parallel jobs are not supported on Spot Instances.
In an unmanaged compute environment, you can manage your own compute resources. This provides more compute resource configuration options, such as using a custom AMI, but you must ensure that your AMI meets the Amazon ECS container instance AMI specification. For more information, see Container Instance AMIs in the Amazon Elastic Container Service Developer Guide . After you have created your unmanaged compute environment, you can use the DescribeComputeEnvironments operation to find the Amazon ECS cluster that is associated with it. Then, manually launch your container instances into that Amazon ECS cluster. For more information, see Launching an Amazon ECS Container Instance in the Amazon Elastic Container Service Developer Guide .
Note
AWS Batch does not upgrade the AMIs in a compute environment after it is created (for example, when a newer version of the Amazon ECS-optimized AMI is available). You are responsible for the management of the guest operating system (including updates and security patches) and any additional application software or utilities that you install on the compute resources. To use a new AMI for your AWS Batch jobs:
Create a new compute environment with the new AMI.
Add the compute environment to an existing job queue.
Remove the old compute environment from your job queue.
Delete the old compute environment.
See also: AWS API Documentation
Request Syntax
client.create_compute_environment(
computeEnvironmentName='string',
type='MANAGED'|'UNMANAGED',
state='ENABLED'|'DISABLED',
computeResources={
'type': 'EC2'|'SPOT',
'allocationStrategy': 'BEST_FIT'|'BEST_FIT_PROGRESSIVE'|'SPOT_CAPACITY_OPTIMIZED',
'minvCpus': 123,
'maxvCpus': 123,
'desiredvCpus': 123,
'instanceTypes': [
'string',
],
'imageId': 'string',
'subnets': [
'string',
],
'securityGroupIds': [
'string',
],
'ec2KeyPair': 'string',
'instanceRole': 'string',
'tags': {
'string': 'string'
},
'placementGroup': 'string',
'bidPercentage': 123,
'spotIamFleetRole': 'string',
'launchTemplate': {
'launchTemplateId': 'string',
'launchTemplateName': 'string',
'version': 'string'
},
'ec2Configuration': [
{
'imageType': 'string',
'imageIdOverride': 'string'
},
]
},
serviceRole='string',
tags={
'string': 'string'
}
)
string
[REQUIRED]
The name for your compute environment. Up to 128 letters (uppercase and lowercase), numbers, hyphens, and underscores are allowed.
string
[REQUIRED]
The type of the compute environment. For more information, see Compute Environments in the AWS Batch User Guide .
string
The state of the compute environment. If the state is ENABLED , then the compute environment accepts jobs from a queue and can scale out automatically based on queues.
dict
Details of the compute resources managed by the compute environment. This parameter is required for managed compute environments. For more information, see Compute Environments in the AWS Batch User Guide .
type (string) -- [REQUIRED]
The type of compute environment: EC2 or SPOT .
allocationStrategy (string) --
The allocation strategy to use for the compute resource in case not enough instances of the best fitting instance type can be allocated. This could be due to availability of the instance type in the region or Amazon EC2 service limits . If this is not specified, the default is BEST_FIT , which will use only the best fitting instance type, waiting for additional capacity if it's not available. This allocation strategy keeps costs lower but can limit scaling. If you are using Spot Fleets with BEST_FIT then the Spot Fleet IAM Role must be specified. BEST_FIT_PROGRESSIVE will select additional instance types that are large enough to meet the requirements of the jobs in the queue, with a preference for instance types with a lower cost per vCPU. SPOT_CAPACITY_OPTIMIZED is only available for Spot Instance compute resources and will select additional instance types that are large enough to meet the requirements of the jobs in the queue, with a preference for instance types that are less likely to be interrupted. For more information, see Allocation Strategies in the AWS Batch User Guide .
minvCpus (integer) -- [REQUIRED]
The minimum number of Amazon EC2 vCPUs that an environment should maintain (even if the compute environment is DISABLED ).
maxvCpus (integer) -- [REQUIRED]
The maximum number of Amazon EC2 vCPUs that an environment can reach.
desiredvCpus (integer) --
The desired number of Amazon EC2 vCPUS in the compute environment.
instanceTypes (list) -- [REQUIRED]
The instances types that may be launched. You can specify instance families to launch any instance type within those families (for example, c5 or p3 ), or you can specify specific sizes within a family (such as c5.8xlarge ). You can also choose optimal to pick instance types (from the C, M, and R instance families) on the fly that match the demand of your job queues.
(string) --
imageId (string) --
The Amazon Machine Image (AMI) ID used for instances launched in the compute environment. This parameter is overridden by the imageIdOverride member of the Ec2Configuration structure.
subnets (list) -- [REQUIRED]
The VPC subnets into which the compute resources are launched. For more information, see VPCs and Subnets in the Amazon VPC User Guide .
(string) --
securityGroupIds (list) --
The Amazon EC2 security groups associated with instances launched in the compute environment. One or more security groups must be specified, either in securityGroupIds or using a launch template referenced in launchTemplate . If security groups are specified using both securityGroupIds and launchTemplate , the values in securityGroupIds will be used.
(string) --
ec2KeyPair (string) --
The Amazon EC2 key pair that is used for instances launched in the compute environment.
instanceRole (string) -- [REQUIRED]
The Amazon ECS instance profile applied to Amazon EC2 instances in a compute environment. You can specify the short name or full Amazon Resource Name (ARN) of an instance profile. For example, `` ecsInstanceRole `` or ``arn:aws:iam::<aws_account_id> :instance-profile/ecsInstanceRole `` . For more information, see Amazon ECS Instance Role in the AWS Batch User Guide .
tags (dict) --
Key-value pair tags to be applied to resources that are launched in the compute environment. For AWS Batch, these take the form of "String1": "String2", where String1 is the tag key and String2 is the tag value—for example, { "Name": "AWS Batch Instance - C4OnDemand" }. These tags can not be updated or removed after the compute environment has been created; any changes require creating a new compute environment and removing the old compute environment. These tags are not seen when using the AWS Batch ListTagsForResource API operation.
(string) --
(string) --
placementGroup (string) --
The Amazon EC2 placement group to associate with your compute resources. If you intend to submit multi-node parallel jobs to your compute environment, you should consider creating a cluster placement group and associate it with your compute resources. This keeps your multi-node parallel job on a logical grouping of instances within a single Availability Zone with high network flow potential. For more information, see Placement Groups in the Amazon EC2 User Guide for Linux Instances .
bidPercentage (integer) --
The maximum percentage that a Spot Instance price can be when compared with the On-Demand price for that instance type before instances are launched. For example, if your maximum percentage is 20%, then the Spot price must be below 20% of the current On-Demand price for that Amazon EC2 instance. You always pay the lowest (market) price and never more than your maximum percentage. If you leave this field empty, the default value is 100% of the On-Demand price.
spotIamFleetRole (string) --
The Amazon Resource Name (ARN) of the Amazon EC2 Spot Fleet IAM role applied to a SPOT compute environment. This role is required if the allocation strategy set to BEST_FIT or if the allocation strategy is not specified. For more information, see Amazon EC2 Spot Fleet Role in the AWS Batch User Guide .
launchTemplate (dict) --
The launch template to use for your compute resources. Any other compute resource parameters that you specify in a CreateComputeEnvironment API operation override the same parameters in the launch template. You must specify either the launch template ID or launch template name in the request, but not both. For more information, see Launch Template Support in the AWS Batch User Guide .
launchTemplateId (string) --
The ID of the launch template.
launchTemplateName (string) --
The name of the launch template.
version (string) --
The version number of the launch template, $Latest , or $Default .
If the value is $Latest , the latest version of the launch template is used. If the value is $Default , the default version of the launch template is used.
Default: $Default .
ec2Configuration (list) --
Provides additional details used to selecting the AMI to use for instances in a compute environment.
(dict) --
Provides information used to select Amazon Machine Images (AMIs) for instances in the compute environment. If the Ec2Configuration is not specified, the default is ECS_AL1 .
imageType (string) -- [REQUIRED]
The image type to match with the instance type to pick an AMI. If the imageIdOverride parameter is not specified, then a recent Amazon ECS-optimized AMI will be used.
ECS_AL2
Amazon Linux 2 − Default for all AWS Graviton-based instance families (for example, C6g , M6g , R6g , and T4g ) and can be used for all non-GPU instance types.
ECS_AL2_NVIDIA
Amazon Linux 2 (GPU) −Default for all GPU instance families (for example P4 and G4 ) and can be used for all non-AWS Graviton-based instance types.
ECS_AL1
Amazon Linux −Default for all non-GPU, non-AWS-Graviton instance families. Amazon Linux is reaching the end-of-life of standard support. For more information, see Amazon Linux AMI .
imageIdOverride (string) --
The AMI ID used for instances launched in the compute environment that match the image type. This setting overrides the imageId set in the computeResource object.
string
[REQUIRED]
The full Amazon Resource Name (ARN) of the IAM role that allows AWS Batch to make calls to other AWS services on your behalf.
If your specified role has a path other than / , then you must either specify the full role ARN (this is recommended) or prefix the role name with the path.
Note
Depending on how you created your AWS Batch service role, its ARN may contain the service-role path prefix. When you only specify the name of the service role, AWS Batch assumes that your ARN does not use the service-role path prefix. Because of this, we recommend that you specify the full ARN of your service role when you create compute environments.
dict
The tags that you apply to the compute environment to help you categorize and organize your resources. Each tag consists of a key and an optional value. For more information, see Tagging AWS Resources in AWS General Reference .
These tags can be updated or removed using the TagResource and UntagResource API operations. These tags do not propagate to the underlying compute resources.
(string) --
(string) --
dict
Response Syntax
{
'computeEnvironmentName': 'string',
'computeEnvironmentArn': 'string'
}
Response Structure
(dict) --
computeEnvironmentName (string) --
The name of the compute environment.
computeEnvironmentArn (string) --
The Amazon Resource Name (ARN) of the compute environment.
{'computeEnvironments': {'computeResources': {'ec2Configuration': [{'imageIdOverride': 'string',
'imageType': 'string'}]}}}
Describes one or more of your compute environments.
If you are using an unmanaged compute environment, you can use the DescribeComputeEnvironment operation to determine the ecsClusterArn that you should launch your Amazon ECS container instances into.
See also: AWS API Documentation
Request Syntax
client.describe_compute_environments(
computeEnvironments=[
'string',
],
maxResults=123,
nextToken='string'
)
list
A list of up to 100 compute environment names or full Amazon Resource Name (ARN) entries.
(string) --
integer
The maximum number of cluster results returned by DescribeComputeEnvironments in paginated output. When this parameter is used, DescribeComputeEnvironments only returns maxResults results in a single page along with a nextToken response element. The remaining results of the initial request can be seen by sending another DescribeComputeEnvironments request with the returned nextToken value. This value can be between 1 and 100. If this parameter is not used, then DescribeComputeEnvironments returns up to 100 results and a nextToken value if applicable.
string
The nextToken value returned from a previous paginated DescribeComputeEnvironments request where maxResults was used and the results exceeded the value of that parameter. Pagination continues from the end of the previous results that returned the nextToken value. This value is null when there are no more results to return.
Note
This token should be treated as an opaque identifier that is only used to retrieve the next items in a list and not for other programmatic purposes.
dict
Response Syntax
{
'computeEnvironments': [
{
'computeEnvironmentName': 'string',
'computeEnvironmentArn': 'string',
'ecsClusterArn': 'string',
'tags': {
'string': 'string'
},
'type': 'MANAGED'|'UNMANAGED',
'state': 'ENABLED'|'DISABLED',
'status': 'CREATING'|'UPDATING'|'DELETING'|'DELETED'|'VALID'|'INVALID',
'statusReason': 'string',
'computeResources': {
'type': 'EC2'|'SPOT',
'allocationStrategy': 'BEST_FIT'|'BEST_FIT_PROGRESSIVE'|'SPOT_CAPACITY_OPTIMIZED',
'minvCpus': 123,
'maxvCpus': 123,
'desiredvCpus': 123,
'instanceTypes': [
'string',
],
'imageId': 'string',
'subnets': [
'string',
],
'securityGroupIds': [
'string',
],
'ec2KeyPair': 'string',
'instanceRole': 'string',
'tags': {
'string': 'string'
},
'placementGroup': 'string',
'bidPercentage': 123,
'spotIamFleetRole': 'string',
'launchTemplate': {
'launchTemplateId': 'string',
'launchTemplateName': 'string',
'version': 'string'
},
'ec2Configuration': [
{
'imageType': 'string',
'imageIdOverride': 'string'
},
]
},
'serviceRole': 'string'
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
computeEnvironments (list) --
The list of compute environments.
(dict) --
An object representing an AWS Batch compute environment.
computeEnvironmentName (string) --
The name of the compute environment.
computeEnvironmentArn (string) --
The Amazon Resource Name (ARN) of the compute environment.
ecsClusterArn (string) --
The Amazon Resource Name (ARN) of the underlying Amazon ECS cluster used by the compute environment.
tags (dict) --
The tags applied to the compute environment.
(string) --
(string) --
type (string) --
The type of the compute environment.
state (string) --
The state of the compute environment. The valid values are ENABLED or DISABLED .
If the state is ENABLED , then the AWS Batch scheduler can attempt to place jobs from an associated job queue on the compute resources within the environment. If the compute environment is managed, then it can scale its instances out or in automatically, based on the job queue demand.
If the state is DISABLED , then the AWS Batch scheduler does not attempt to place jobs within the environment. Jobs in a STARTING or RUNNING state continue to progress normally. Managed compute environments in the DISABLED state do not scale out. However, they scale in to minvCpus value after instances become idle.
status (string) --
The current status of the compute environment (for example, CREATING or VALID ).
statusReason (string) --
A short, human-readable string to provide additional details about the current status of the compute environment.
computeResources (dict) --
The compute resources defined for the compute environment.
type (string) --
The type of compute environment: EC2 or SPOT .
allocationStrategy (string) --
The allocation strategy to use for the compute resource in case not enough instances of the best fitting instance type can be allocated. This could be due to availability of the instance type in the region or Amazon EC2 service limits . If this is not specified, the default is BEST_FIT , which will use only the best fitting instance type, waiting for additional capacity if it's not available. This allocation strategy keeps costs lower but can limit scaling. If you are using Spot Fleets with BEST_FIT then the Spot Fleet IAM Role must be specified. BEST_FIT_PROGRESSIVE will select additional instance types that are large enough to meet the requirements of the jobs in the queue, with a preference for instance types with a lower cost per vCPU. SPOT_CAPACITY_OPTIMIZED is only available for Spot Instance compute resources and will select additional instance types that are large enough to meet the requirements of the jobs in the queue, with a preference for instance types that are less likely to be interrupted. For more information, see Allocation Strategies in the AWS Batch User Guide .
minvCpus (integer) --
The minimum number of Amazon EC2 vCPUs that an environment should maintain (even if the compute environment is DISABLED ).
maxvCpus (integer) --
The maximum number of Amazon EC2 vCPUs that an environment can reach.
desiredvCpus (integer) --
The desired number of Amazon EC2 vCPUS in the compute environment.
instanceTypes (list) --
The instances types that may be launched. You can specify instance families to launch any instance type within those families (for example, c5 or p3 ), or you can specify specific sizes within a family (such as c5.8xlarge ). You can also choose optimal to pick instance types (from the C, M, and R instance families) on the fly that match the demand of your job queues.
(string) --
imageId (string) --
The Amazon Machine Image (AMI) ID used for instances launched in the compute environment. This parameter is overridden by the imageIdOverride member of the Ec2Configuration structure.
subnets (list) --
The VPC subnets into which the compute resources are launched. For more information, see VPCs and Subnets in the Amazon VPC User Guide .
(string) --
securityGroupIds (list) --
The Amazon EC2 security groups associated with instances launched in the compute environment. One or more security groups must be specified, either in securityGroupIds or using a launch template referenced in launchTemplate . If security groups are specified using both securityGroupIds and launchTemplate , the values in securityGroupIds will be used.
(string) --
ec2KeyPair (string) --
The Amazon EC2 key pair that is used for instances launched in the compute environment.
instanceRole (string) --
The Amazon ECS instance profile applied to Amazon EC2 instances in a compute environment. You can specify the short name or full Amazon Resource Name (ARN) of an instance profile. For example, `` ecsInstanceRole `` or ``arn:aws:iam::<aws_account_id> :instance-profile/ecsInstanceRole `` . For more information, see Amazon ECS Instance Role in the AWS Batch User Guide .
tags (dict) --
Key-value pair tags to be applied to resources that are launched in the compute environment. For AWS Batch, these take the form of "String1": "String2", where String1 is the tag key and String2 is the tag value—for example, { "Name": "AWS Batch Instance - C4OnDemand" }. These tags can not be updated or removed after the compute environment has been created; any changes require creating a new compute environment and removing the old compute environment. These tags are not seen when using the AWS Batch ListTagsForResource API operation.
(string) --
(string) --
placementGroup (string) --
The Amazon EC2 placement group to associate with your compute resources. If you intend to submit multi-node parallel jobs to your compute environment, you should consider creating a cluster placement group and associate it with your compute resources. This keeps your multi-node parallel job on a logical grouping of instances within a single Availability Zone with high network flow potential. For more information, see Placement Groups in the Amazon EC2 User Guide for Linux Instances .
bidPercentage (integer) --
The maximum percentage that a Spot Instance price can be when compared with the On-Demand price for that instance type before instances are launched. For example, if your maximum percentage is 20%, then the Spot price must be below 20% of the current On-Demand price for that Amazon EC2 instance. You always pay the lowest (market) price and never more than your maximum percentage. If you leave this field empty, the default value is 100% of the On-Demand price.
spotIamFleetRole (string) --
The Amazon Resource Name (ARN) of the Amazon EC2 Spot Fleet IAM role applied to a SPOT compute environment. This role is required if the allocation strategy set to BEST_FIT or if the allocation strategy is not specified. For more information, see Amazon EC2 Spot Fleet Role in the AWS Batch User Guide .
launchTemplate (dict) --
The launch template to use for your compute resources. Any other compute resource parameters that you specify in a CreateComputeEnvironment API operation override the same parameters in the launch template. You must specify either the launch template ID or launch template name in the request, but not both. For more information, see Launch Template Support in the AWS Batch User Guide .
launchTemplateId (string) --
The ID of the launch template.
launchTemplateName (string) --
The name of the launch template.
version (string) --
The version number of the launch template, $Latest , or $Default .
If the value is $Latest , the latest version of the launch template is used. If the value is $Default , the default version of the launch template is used.
Default: $Default .
ec2Configuration (list) --
Provides additional details used to selecting the AMI to use for instances in a compute environment.
(dict) --
Provides information used to select Amazon Machine Images (AMIs) for instances in the compute environment. If the Ec2Configuration is not specified, the default is ECS_AL1 .
imageType (string) --
The image type to match with the instance type to pick an AMI. If the imageIdOverride parameter is not specified, then a recent Amazon ECS-optimized AMI will be used.
ECS_AL2
Amazon Linux 2 − Default for all AWS Graviton-based instance families (for example, C6g , M6g , R6g , and T4g ) and can be used for all non-GPU instance types.
ECS_AL2_NVIDIA
Amazon Linux 2 (GPU) −Default for all GPU instance families (for example P4 and G4 ) and can be used for all non-AWS Graviton-based instance types.
ECS_AL1
Amazon Linux −Default for all non-GPU, non-AWS-Graviton instance families. Amazon Linux is reaching the end-of-life of standard support. For more information, see Amazon Linux AMI .
imageIdOverride (string) --
The AMI ID used for instances launched in the compute environment that match the image type. This setting overrides the imageId set in the computeResource object.
serviceRole (string) --
The service role associated with the compute environment that allows AWS Batch to make calls to AWS API operations on your behalf.
nextToken (string) --
The nextToken value to include in a future DescribeComputeEnvironments request. When the results of a DescribeJobDefinitions request exceed maxResults , this value can be used to retrieve the next page of results. This value is null when there are no more results to return.