AWS API Changes

2026/07/27 - Amazon EMR Containers - 1 new 3 updated api methods

Changes With this launch, you can now set concurrent job limits on a virtual cluster, giving you fine-grained control over how many job runs execute at once and how many can wait in queue.

UpdateVirtualCluster (new)

Link ¶

Updates a virtual cluster. Virtual cluster is a managed entity on Amazon EMR on EKS. You can create, update, describe, list and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements.

See also: AWS API Documentation

Request Syntax

client.update_virtual_cluster(
    id='string',
    schedulerConfiguration={
        'maxInQueueJobRuns': 123,
        'maxConcurrentJobRuns': 123
    },
    clientToken='string'
)

type id:

string

param id:

[REQUIRED]

The ID of the virtual cluster to update.

type schedulerConfiguration:

dict

param schedulerConfiguration:

The scheduler configuration to apply to the virtual cluster. The new configuration fully replaces the existing one. If you omit a field, the corresponding limit is removed.

maxInQueueJobRuns (integer) --

The maximum number of job runs that can be in the PENDING or SUBMITTED state at any time for the virtual cluster. When the queue is full, the service rejects StartJobRun requests with a ValidationException. If you omit this field, the service applies no queue-depth limit.
maxConcurrentJobRuns (integer) --

The maximum number of job runs that can be in the RUNNING state at any time for the virtual cluster. As running slots free up, queued job runs start automatically. If you omit this field, the service applies no concurrency limit.

type clientToken:

string

param clientToken:

[REQUIRED]

A unique, case-sensitive identifier that you provide to ensure that the operation completes no more than one time. If this token matches a previous request, the service ignores the request, but does not return an error.

This field is autopopulated if not provided.

rtype:

dict

returns:

Response Syntax

{
    'virtualCluster': {
        'id': 'string',
        'name': 'string',
        'arn': 'string',
        'state': 'RUNNING'|'TERMINATING'|'TERMINATED'|'ARRESTED',
        'containerProvider': {
            'type': 'EKS',
            'id': 'string',
            'info': {
                'eksInfo': {
                    'namespace': 'string',
                    'nodeLabel': 'string'
                }
            }
        },
        'createdAt': datetime(2015, 1, 1),
        'tags': {
            'string': 'string'
        },
        'securityConfigurationId': 'string',
        'sessionEnabled': True|False,
        'schedulerConfiguration': {
            'maxInQueueJobRuns': 123,
            'maxConcurrentJobRuns': 123
        },
        'schedulerStatus': {
            'currentInQueueJobRuns': 123,
            'currentConcurrentJobRuns': 123
        }
    }
}

Response Structure

(dict) --

Contains the virtual cluster returned after a successful update request.
- virtualCluster (dict) --
  
  The updated virtual cluster.
  - id (string) --
    
    The ID of the virtual cluster.
  - name (string) --
    
    The name of the virtual cluster.
  - arn (string) --
    
    The ARN of the virtual cluster.
  - state (string) --
    
    The state of the virtual cluster.
  - containerProvider (dict) --
    
    The container provider of the virtual cluster.
    - type (string) --
      
      The type of the container provider. Amazon EKS is the only supported type as of now.
    - id (string) --
      
      The ID of the container cluster.
    - info (dict) --
      
      The information about the container cluster.
      Note
      
      This is a Tagged Union structure. Only one of the following top level keys will be set: eksInfo. If a client receives an unknown member it will set SDK_UNKNOWN_MEMBER as the top level key, which maps to the name or tag of the unknown member. The structure of SDK_UNKNOWN_MEMBER is as follows:
      
      'SDK_UNKNOWN_MEMBER': {'name': 'UnknownMemberName'}
      - eksInfo (dict) --
        
        The information about the Amazon EKS cluster.
        
        namespace (string) --
        
        The namespaces of the Amazon EKS cluster.
        
        nodeLabel (string) --
        
        The nodeLabel of the nodes where the resources of this virtual cluster can get scheduled. It requires relevant scaling and policy engine addons.
  - createdAt (datetime) --
    
    The date and time when the virtual cluster is created.
  - tags (dict) --
    
    The assigned tags of the virtual cluster.
    - (string) --
      - (string) --
  - securityConfigurationId (string) --
    
    The ID of the security configuration.
  - sessionEnabled (boolean) --
    
    Specifies whether the virtual cluster has session support enabled.
  - schedulerConfiguration (dict) --
    
    The scheduler configuration (concurrency and queue limits) applied to the virtual cluster. The service does not return this field when no scheduler limits are configured.
    - maxInQueueJobRuns (integer) --
      
      The maximum number of job runs that can be in the PENDING or SUBMITTED state at any time for the virtual cluster. When the queue is full, the service rejects StartJobRun requests with a ValidationException. If you omit this field, the service applies no queue-depth limit.
    - maxConcurrentJobRuns (integer) --
      
      The maximum number of job runs that can be in the RUNNING state at any time for the virtual cluster. As running slots free up, queued job runs start automatically. If you omit this field, the service applies no concurrency limit.
  - schedulerStatus (dict) --
    
    The current in-queue and concurrent job-run counts for the virtual cluster.
    - currentInQueueJobRuns (integer) --
      
      The number of job runs currently waiting in the queue ( PENDING or SUBMITTED) for the virtual cluster.
    - currentConcurrentJobRuns (integer) --
      
      The number of job runs currently in the RUNNING state for the virtual cluster.

CreateVirtualCluster (updated)

Link ¶
Changes (request)

{'schedulerConfiguration': {'maxConcurrentJobRuns': 'integer',
                            'maxInQueueJobRuns': 'integer'}}

Creates a virtual cluster. Virtual cluster is a managed entity on Amazon EMR on EKS. You can create, update, describe, list and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements.

See also: AWS API Documentation

Request Syntax

client.create_virtual_cluster(
    name='string',
    containerProvider={
        'type': 'EKS',
        'id': 'string',
        'info': {
            'eksInfo': {
                'namespace': 'string',
                'nodeLabel': 'string'
            }
        }
    },
    clientToken='string',
    tags={
        'string': 'string'
    },
    securityConfigurationId='string',
    sessionEnabled=True|False,
    schedulerConfiguration={
        'maxInQueueJobRuns': 123,
        'maxConcurrentJobRuns': 123
    }
)

type name:

string

param name:

[REQUIRED]

The specified name of the virtual cluster.

type containerProvider:

dict

param containerProvider:

[REQUIRED]

The container provider of the virtual cluster.

type (string) -- [REQUIRED]

The type of the container provider. Amazon EKS is the only supported type as of now.
id (string) -- [REQUIRED]

The ID of the container cluster.
info (dict) --

The information about the container cluster.

Note

This is a Tagged Union structure. Only one of the following top level keys can be set: eksInfo.
- eksInfo (dict) --
  
  The information about the Amazon EKS cluster.
  - namespace (string) --
    
    The namespaces of the Amazon EKS cluster.
  - nodeLabel (string) --
    
    The nodeLabel of the nodes where the resources of this virtual cluster can get scheduled. It requires relevant scaling and policy engine addons.

type clientToken:

string

param clientToken:

[REQUIRED]

The client token of the virtual cluster.

This field is autopopulated if not provided.

type tags:

dict

param tags:

The tags assigned to the virtual cluster.

(string) --
- (string) --

type securityConfigurationId:

string

param securityConfigurationId:

The ID of the security configuration.

type sessionEnabled:

boolean

param sessionEnabled:

Indicates whether the virtual cluster has session support enabled.

type schedulerConfiguration:

dict

param schedulerConfiguration:

The scheduler configuration (concurrency and queue limits) to apply to the virtual cluster at creation time. When omitted, no limits are applied.

maxInQueueJobRuns (integer) --

The maximum number of job runs that can be in the PENDING or SUBMITTED state at any time for the virtual cluster. When the queue is full, the service rejects StartJobRun requests with a ValidationException. If you omit this field, the service applies no queue-depth limit.
maxConcurrentJobRuns (integer) --

The maximum number of job runs that can be in the RUNNING state at any time for the virtual cluster. As running slots free up, queued job runs start automatically. If you omit this field, the service applies no concurrency limit.

rtype:

dict

returns:

Response Syntax

{
    'id': 'string',
    'name': 'string',
    'arn': 'string'
}

Response Structure

(dict) --
- id (string) --
  
  This output contains the virtual cluster ID.
- name (string) --
  
  This output contains the name of the virtual cluster.
- arn (string) --
  
  This output contains the ARN of virtual cluster.

DescribeVirtualCluster (updated)

Link ¶
Changes (response)

{'virtualCluster': {'schedulerConfiguration': {'maxConcurrentJobRuns': 'integer',
                                               'maxInQueueJobRuns': 'integer'},
                    'schedulerStatus': {'currentConcurrentJobRuns': 'integer',
                                        'currentInQueueJobRuns': 'integer'}}}

Displays detailed information about a specified virtual cluster. Virtual cluster is a managed entity on Amazon EMR on EKS. You can create, update, describe, list and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements.

See also: AWS API Documentation

Request Syntax

client.describe_virtual_cluster(
    id='string'
)

type id:

string

param id:

[REQUIRED]

The ID of the virtual cluster that will be described.

rtype:

dict

returns:

Response Syntax

{
    'virtualCluster': {
        'id': 'string',
        'name': 'string',
        'arn': 'string',
        'state': 'RUNNING'|'TERMINATING'|'TERMINATED'|'ARRESTED',
        'containerProvider': {
            'type': 'EKS',
            'id': 'string',
            'info': {
                'eksInfo': {
                    'namespace': 'string',
                    'nodeLabel': 'string'
                }
            }
        },
        'createdAt': datetime(2015, 1, 1),
        'tags': {
            'string': 'string'
        },
        'securityConfigurationId': 'string',
        'sessionEnabled': True|False,
        'schedulerConfiguration': {
            'maxInQueueJobRuns': 123,
            'maxConcurrentJobRuns': 123
        },
        'schedulerStatus': {
            'currentInQueueJobRuns': 123,
            'currentConcurrentJobRuns': 123
        }
    }
}

Response Structure

(dict) --
- virtualCluster (dict) --
  
  This output displays information about the specified virtual cluster.
  - id (string) --
    
    The ID of the virtual cluster.
  - name (string) --
    
    The name of the virtual cluster.
  - arn (string) --
    
    The ARN of the virtual cluster.
  - state (string) --
    
    The state of the virtual cluster.
  - containerProvider (dict) --
    
    The container provider of the virtual cluster.
    - type (string) --
      
      The type of the container provider. Amazon EKS is the only supported type as of now.
    - id (string) --
      
      The ID of the container cluster.
    - info (dict) --
      
      The information about the container cluster.
      Note
      
      This is a Tagged Union structure. Only one of the following top level keys will be set: eksInfo. If a client receives an unknown member it will set SDK_UNKNOWN_MEMBER as the top level key, which maps to the name or tag of the unknown member. The structure of SDK_UNKNOWN_MEMBER is as follows:
      
      'SDK_UNKNOWN_MEMBER': {'name': 'UnknownMemberName'}
      - eksInfo (dict) --
        
        The information about the Amazon EKS cluster.
        
        namespace (string) --
        
        The namespaces of the Amazon EKS cluster.
        
        nodeLabel (string) --
        
        The nodeLabel of the nodes where the resources of this virtual cluster can get scheduled. It requires relevant scaling and policy engine addons.
  - createdAt (datetime) --
    
    The date and time when the virtual cluster is created.
  - tags (dict) --
    
    The assigned tags of the virtual cluster.
    - (string) --
      - (string) --
  - securityConfigurationId (string) --
    
    The ID of the security configuration.
  - sessionEnabled (boolean) --
    
    Specifies whether the virtual cluster has session support enabled.
  - schedulerConfiguration (dict) --
    
    The scheduler configuration (concurrency and queue limits) applied to the virtual cluster. The service does not return this field when no scheduler limits are configured.
    - maxInQueueJobRuns (integer) --
      
      The maximum number of job runs that can be in the PENDING or SUBMITTED state at any time for the virtual cluster. When the queue is full, the service rejects StartJobRun requests with a ValidationException. If you omit this field, the service applies no queue-depth limit.
    - maxConcurrentJobRuns (integer) --
      
      The maximum number of job runs that can be in the RUNNING state at any time for the virtual cluster. As running slots free up, queued job runs start automatically. If you omit this field, the service applies no concurrency limit.
  - schedulerStatus (dict) --
    
    The current in-queue and concurrent job-run counts for the virtual cluster.
    - currentInQueueJobRuns (integer) --
      
      The number of job runs currently waiting in the queue ( PENDING or SUBMITTED) for the virtual cluster.
    - currentConcurrentJobRuns (integer) --
      
      The number of job runs currently in the RUNNING state for the virtual cluster.

ListVirtualClusters (updated)

Link ¶
Changes (response)

{'virtualClusters': {'schedulerConfiguration': {'maxConcurrentJobRuns': 'integer',
                                                'maxInQueueJobRuns': 'integer'},
                     'schedulerStatus': {'currentConcurrentJobRuns': 'integer',
                                         'currentInQueueJobRuns': 'integer'}}}

Lists information about the specified virtual cluster. Virtual cluster is a managed entity on Amazon EMR on EKS. You can create, update, describe, list and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements.

See also: AWS API Documentation

Request Syntax

client.list_virtual_clusters(
    containerProviderId='string',
    containerProviderType='EKS',
    createdAfter=datetime(2015, 1, 1),
    createdBefore=datetime(2015, 1, 1),
    states=[
        'RUNNING'|'TERMINATING'|'TERMINATED'|'ARRESTED',
    ],
    maxResults=123,
    nextToken='string',
    eksAccessEntryIntegrated=True|False
)

type containerProviderId:

string

param containerProviderId:

The container provider ID of the virtual cluster.

type containerProviderType:

string

param containerProviderType:

The container provider type of the virtual cluster. Amazon EKS is the only supported type as of now.

type createdAfter:

datetime

param createdAfter:

The date and time after which the virtual clusters are created.

type createdBefore:

datetime

param createdBefore:

The date and time before which the virtual clusters are created.

type states:

list

param states:

The states of the requested virtual clusters.

(string) --

type maxResults:

integer

param maxResults:

The maximum number of virtual clusters that can be listed.

type nextToken:

string

param nextToken:

The token for the next set of virtual clusters to return.

type eksAccessEntryIntegrated:

boolean

param eksAccessEntryIntegrated:

Optional Boolean that specifies whether the operation should return the virtual clusters that have the access entry integration enabled or disabled. If not specified, the operation returns all applicable virtual clusters.

rtype:

dict

returns:

Response Syntax

{
    'virtualClusters': [
        {
            'id': 'string',
            'name': 'string',
            'arn': 'string',
            'state': 'RUNNING'|'TERMINATING'|'TERMINATED'|'ARRESTED',
            'containerProvider': {
                'type': 'EKS',
                'id': 'string',
                'info': {
                    'eksInfo': {
                        'namespace': 'string',
                        'nodeLabel': 'string'
                    }
                }
            },
            'createdAt': datetime(2015, 1, 1),
            'tags': {
                'string': 'string'
            },
            'securityConfigurationId': 'string',
            'sessionEnabled': True|False,
            'schedulerConfiguration': {
                'maxInQueueJobRuns': 123,
                'maxConcurrentJobRuns': 123
            },
            'schedulerStatus': {
                'currentInQueueJobRuns': 123,
                'currentConcurrentJobRuns': 123
            }
        },
    ],
    'nextToken': 'string'
}

Response Structure

(dict) --
- virtualClusters (list) --
  
  This output lists the specified virtual clusters.
  - (dict) --
    
    This entity describes a virtual cluster. A virtual cluster is a Kubernetes namespace that Amazon EMR is registered with. Amazon EMR uses virtual clusters to run jobs and host endpoints. Multiple virtual clusters can be backed by the same physical cluster. However, each virtual cluster maps to one namespace on an Amazon EKS cluster. Virtual clusters do not create any active resources that contribute to your bill or that require lifecycle management outside the service.
    - id (string) --
      
      The ID of the virtual cluster.
    - name (string) --
      
      The name of the virtual cluster.
    - arn (string) --
      
      The ARN of the virtual cluster.
    - state (string) --
      
      The state of the virtual cluster.
    - containerProvider (dict) --
      
      The container provider of the virtual cluster.
      - type (string) --
        
        The type of the container provider. Amazon EKS is the only supported type as of now.
      - id (string) --
        
        The ID of the container cluster.
      - info (dict) --
        
        The information about the container cluster.
        
        Note
        
        This is a Tagged Union structure. Only one of the following top level keys will be set: eksInfo. If a client receives an unknown member it will set SDK_UNKNOWN_MEMBER as the top level key, which maps to the name or tag of the unknown member. The structure of SDK_UNKNOWN_MEMBER is as follows:
        
        'SDK_UNKNOWN_MEMBER': {'name': 'UnknownMemberName'}
        
        eksInfo (dict) --
        
        The information about the Amazon EKS cluster.
        
        namespace (string) --
        
        The namespaces of the Amazon EKS cluster.
        
        nodeLabel (string) --
        
        The nodeLabel of the nodes where the resources of this virtual cluster can get scheduled. It requires relevant scaling and policy engine addons.
    - createdAt (datetime) --
      
      The date and time when the virtual cluster is created.
    - tags (dict) --
      
      The assigned tags of the virtual cluster.
      - (string) --
        
        (string) --
    - securityConfigurationId (string) --
      
      The ID of the security configuration.
    - sessionEnabled (boolean) --
      
      Specifies whether the virtual cluster has session support enabled.
    - schedulerConfiguration (dict) --
      
      The scheduler configuration (concurrency and queue limits) applied to the virtual cluster. The service does not return this field when no scheduler limits are configured.
      - maxInQueueJobRuns (integer) --
        
        The maximum number of job runs that can be in the PENDING or SUBMITTED state at any time for the virtual cluster. When the queue is full, the service rejects StartJobRun requests with a ValidationException. If you omit this field, the service applies no queue-depth limit.
      - maxConcurrentJobRuns (integer) --
        
        The maximum number of job runs that can be in the RUNNING state at any time for the virtual cluster. As running slots free up, queued job runs start automatically. If you omit this field, the service applies no concurrency limit.
    - schedulerStatus (dict) --
      
      The current in-queue and concurrent job-run counts for the virtual cluster.
      - currentInQueueJobRuns (integer) --
        
        The number of job runs currently waiting in the queue ( PENDING or SUBMITTED) for the virtual cluster.
      - currentConcurrentJobRuns (integer) --
        
        The number of job runs currently in the RUNNING state for the virtual cluster.
- nextToken (string) --
  
  This output displays the token for the next set of virtual clusters.