AWS API Changes

2026/09/18 - AWS Glue - 3 updated api methods

Changes Introducing AWS Glue Data Quality advanced rule recommendations for faster recommendations. This capability uses Amazon Athena to analyze a sample of table data and Amazon Bedrock to recommend DQDL rules.

GetDataQualityRuleRecommendationRun (updated)

Link ¶
Changes (response)

{'RecommendationMode': 'BASIC | ADVANCED'}

Gets the specified recommendation run that was used to generate rules.

See also: AWS API Documentation

Request Syntax

client.get_data_quality_rule_recommendation_run(
    RunId='string'
)

type RunId:

string

param RunId:

[REQUIRED]

The unique run identifier associated with this run.

rtype:

dict

returns:

Response Syntax

{
    'RunId': 'string',
    'DataSource': {
        'GlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            }
        },
        'DataQualityGlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            },
            'PreProcessingQuery': 'string'
        }
    },
    'Role': 'string',
    'NumberOfWorkers': 123,
    'Timeout': 123,
    'Status': 'STARTING'|'RUNNING'|'STOPPING'|'STOPPED'|'SUCCEEDED'|'FAILED'|'TIMEOUT',
    'ErrorString': 'string',
    'StartedOn': datetime(2015, 1, 1),
    'LastModifiedOn': datetime(2015, 1, 1),
    'CompletedOn': datetime(2015, 1, 1),
    'ExecutionTime': 123,
    'RecommendedRuleset': 'string',
    'CreatedRulesetName': 'string',
    'DataQualitySecurityConfiguration': 'string',
    'AdditionalRunOptions': {
        'CustomLogGroupPrefix': 'string'
    },
    'RecommendationMode': 'BASIC'|'ADVANCED'
}

Response Structure

(dict) --

The response for the Data Quality rule recommendation run.
- RunId (string) --
  
  The unique run identifier associated with this run.
- DataSource (dict) --
  
  The data source (an Glue table) associated with this run.
  - GlueTable (dict) --
    
    An Glue table.
    - DatabaseName (string) --
      
      A database name in the Glue Data Catalog.
    - TableName (string) --
      
      A table name in the Glue Data Catalog.
    - CatalogId (string) --
      
      A unique identifier for the Glue Data Catalog.
    - ConnectionName (string) --
      
      The name of the connection to the Glue Data Catalog.
    - AdditionalOptions (dict) --
      
      Additional options for the table. Currently there are two keys supported:
      - pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
      - catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
      - (string) --
        
        (string) --
  - DataQualityGlueTable (dict) --
    
    An Glue table for Data Quality Operations.
    - DatabaseName (string) --
      
      A database name in the Glue Data Catalog.
    - TableName (string) --
      
      A table name in the Glue Data Catalog.
    - CatalogId (string) --
      
      A unique identifier for the Glue Data Catalog.
    - ConnectionName (string) --
      
      The name of the connection to the Glue Data Catalog.
    - AdditionalOptions (dict) --
      
      Additional options for the table. Currently there are two keys supported:
      - pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
      - catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
      - (string) --
        
        (string) --
    - PreProcessingQuery (string) --
      
      SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.
- Role (string) --
  
  The IAM role that Glue assumes to access resources for the run.
- NumberOfWorkers (integer) --
  
  The number of G.1X workers to be used in the run. The default is 5.
- Timeout (integer) --
  
  The timeout for a run in minutes. This is the maximum time that a run can consume resources before it is terminated and enters TIMEOUT status. The default is 2,880 minutes (48 hours).
- Status (string) --
  
  The status for this run.
- ErrorString (string) --
  
  The error strings that are associated with the run.
- StartedOn (datetime) --
  
  The date and time when this run started.
- LastModifiedOn (datetime) --
  
  A timestamp. The last point in time when this data quality rule recommendation run was modified.
- CompletedOn (datetime) --
  
  The date and time when this run was completed.
- ExecutionTime (integer) --
  
  The amount of time (in seconds) that the run consumed resources.
- RecommendedRuleset (string) --
  
  When a start rule recommendation run completes, it creates a recommended ruleset (a set of rules). This member has those rules in Data Quality Definition Language (DQDL) format.
- CreatedRulesetName (string) --
  
  The name of the ruleset that was created by the run.
- DataQualitySecurityConfiguration (string) --
  
  The name of the security configuration created with the data quality encryption option.
- AdditionalRunOptions (dict) --
  
  Additional run options you can specify for a recommendation run.
  - CustomLogGroupPrefix (string) --
    
    A custom prefix for the CloudWatch log group names. When specified, recommendation run logs are written to <CustomLogGroupPrefix>/error and <CustomLogGroupPrefix>/output instead of the default /aws-glue/data-quality/error and /aws-glue/data-quality/output log groups.
- RecommendationMode (string) --
  
  The mode that Glue Data Quality uses to recommend rules.
  
  The default is BASIC.

ListDataQualityRuleRecommendationRuns (updated)

Link ¶
Changes (response)

{'Runs': {'RecommendationMode': 'BASIC | ADVANCED'}}

Lists the recommendation runs meeting the filter criteria.

See also: AWS API Documentation

Request Syntax

client.list_data_quality_rule_recommendation_runs(
    Filter={
        'DataSource': {
            'GlueTable': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'CatalogId': 'string',
                'ConnectionName': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                }
            },
            'DataQualityGlueTable': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'CatalogId': 'string',
                'ConnectionName': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'PreProcessingQuery': 'string'
            }
        },
        'StartedBefore': datetime(2015, 1, 1),
        'StartedAfter': datetime(2015, 1, 1)
    },
    NextToken='string',
    MaxResults=123,
    Tags={
        'string': 'string'
    }
)

type Filter:

dict

param Filter:

The filter criteria.

DataSource (dict) -- [REQUIRED]

Filter based on a specified data source (Glue table).
- GlueTable (dict) --
  
  An Glue table.
  - DatabaseName (string) -- [REQUIRED]
    
    A database name in the Glue Data Catalog.
  - TableName (string) -- [REQUIRED]
    
    A table name in the Glue Data Catalog.
  - CatalogId (string) --
    
    A unique identifier for the Glue Data Catalog.
  - ConnectionName (string) --
    
    The name of the connection to the Glue Data Catalog.
  - AdditionalOptions (dict) --
    
    Additional options for the table. Currently there are two keys supported:
    - pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
    - catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
    - (string) --
      - (string) --
- DataQualityGlueTable (dict) --
  
  An Glue table for Data Quality Operations.
  - DatabaseName (string) -- [REQUIRED]
    
    A database name in the Glue Data Catalog.
  - TableName (string) -- [REQUIRED]
    
    A table name in the Glue Data Catalog.
  - CatalogId (string) --
    
    A unique identifier for the Glue Data Catalog.
  - ConnectionName (string) --
    
    The name of the connection to the Glue Data Catalog.
  - AdditionalOptions (dict) --
    
    Additional options for the table. Currently there are two keys supported:
    - pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
    - catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
    - (string) --
      - (string) --
  - PreProcessingQuery (string) --
    
    SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.
StartedBefore (datetime) --

Filter based on time for results started before provided time.
StartedAfter (datetime) --

Filter based on time for results started after provided time.

type NextToken:

string

param NextToken:

A paginated token to offset the results.

type MaxResults:

integer

param MaxResults:

The maximum number of results to return.

type Tags:

dict

param Tags:

A list of key-value pair tags to filter recommendation runs.

(string) --
- (string) --

rtype:

dict

returns:

Response Syntax

{
    'Runs': [
        {
            'RunId': 'string',
            'Status': 'STARTING'|'RUNNING'|'STOPPING'|'STOPPED'|'SUCCEEDED'|'FAILED'|'TIMEOUT',
            'StartedOn': datetime(2015, 1, 1),
            'DataSource': {
                'GlueTable': {
                    'DatabaseName': 'string',
                    'TableName': 'string',
                    'CatalogId': 'string',
                    'ConnectionName': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    }
                },
                'DataQualityGlueTable': {
                    'DatabaseName': 'string',
                    'TableName': 'string',
                    'CatalogId': 'string',
                    'ConnectionName': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'PreProcessingQuery': 'string'
                }
            },
            'CreatedRulesetName': 'string',
            'RecommendationMode': 'BASIC'|'ADVANCED'
        },
    ],
    'NextToken': 'string'
}

Response Structure

(dict) --
- Runs (list) --
  
  A list of DataQualityRuleRecommendationRunDescription objects.
  - (dict) --
    
    Describes the result of a data quality rule recommendation run.
    - RunId (string) --
      
      The unique run identifier associated with this run.
    - Status (string) --
      
      The status for this run.
    - StartedOn (datetime) --
      
      The date and time when this run started.
    - DataSource (dict) --
      
      The data source (Glue table) associated with the recommendation run.
      - GlueTable (dict) --
        
        An Glue table.
        
        DatabaseName (string) --
        
        A database name in the Glue Data Catalog.
        
        TableName (string) --
        
        A table name in the Glue Data Catalog.
        
        CatalogId (string) --
        
        A unique identifier for the Glue Data Catalog.
        
        ConnectionName (string) --
        
        The name of the connection to the Glue Data Catalog.
        
        AdditionalOptions (dict) --
        
        Additional options for the table. Currently there are two keys supported:
        
        pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
        
        catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
        
        (string) --
        
        (string) --
      - DataQualityGlueTable (dict) --
        
        An Glue table for Data Quality Operations.
        
        DatabaseName (string) --
        
        A database name in the Glue Data Catalog.
        
        TableName (string) --
        
        A table name in the Glue Data Catalog.
        
        CatalogId (string) --
        
        A unique identifier for the Glue Data Catalog.
        
        ConnectionName (string) --
        
        The name of the connection to the Glue Data Catalog.
        
        AdditionalOptions (dict) --
        
        Additional options for the table. Currently there are two keys supported:
        
        pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
        
        catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
        
        (string) --
        
        (string) --
        
        PreProcessingQuery (string) --
        
        SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.
    - CreatedRulesetName (string) --
      
      The name of the ruleset that was created by the recommendation run.
    - RecommendationMode (string) --
      
      The mode that Glue Data Quality uses to recommend rules.
      
      The default is BASIC.
- NextToken (string) --
  
  A pagination token, if more results are available.

StartDataQualityRuleRecommendationRun (updated)

Link ¶
Changes (request)

{'RecommendationMode': 'BASIC | ADVANCED'}

Starts a recommendation run that is used to generate rules when you don't know what rules to write. Glue Data Quality analyzes the data and comes up with recommendations for a potential ruleset. You can then triage the ruleset and modify the generated ruleset to your liking.

Recommendation runs are automatically deleted after 90 days.

See also: AWS API Documentation

Request Syntax

client.start_data_quality_rule_recommendation_run(
    DataSource={
        'GlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            }
        },
        'DataQualityGlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            },
            'PreProcessingQuery': 'string'
        }
    },
    Role='string',
    NumberOfWorkers=123,
    Timeout=123,
    CreatedRulesetName='string',
    DataQualitySecurityConfiguration='string',
    ClientToken='string',
    AdditionalRunOptions={
        'CustomLogGroupPrefix': 'string'
    },
    RecommendationMode='BASIC'|'ADVANCED'
)

type DataSource:

dict

param DataSource:

[REQUIRED]

The data source (Glue table) associated with this run.

GlueTable (dict) --

An Glue table.
- DatabaseName (string) -- [REQUIRED]
  
  A database name in the Glue Data Catalog.
- TableName (string) -- [REQUIRED]
  
  A table name in the Glue Data Catalog.
- CatalogId (string) --
  
  A unique identifier for the Glue Data Catalog.
- ConnectionName (string) --
  
  The name of the connection to the Glue Data Catalog.
- AdditionalOptions (dict) --
  
  Additional options for the table. Currently there are two keys supported:
  - pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
  - catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
  - (string) --
    - (string) --
DataQualityGlueTable (dict) --

An Glue table for Data Quality Operations.
- DatabaseName (string) -- [REQUIRED]
  
  A database name in the Glue Data Catalog.
- TableName (string) -- [REQUIRED]
  
  A table name in the Glue Data Catalog.
- CatalogId (string) --
  
  A unique identifier for the Glue Data Catalog.
- ConnectionName (string) --
  
  The name of the connection to the Glue Data Catalog.
- AdditionalOptions (dict) --
  
  Additional options for the table. Currently there are two keys supported:
  - pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
  - catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
  - (string) --
    - (string) --
- PreProcessingQuery (string) --
  
  SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.

type Role:

string

param Role:

[REQUIRED]

The IAM role that Glue assumes to access resources for the run.

For more information, see Configure IAM permissions for Glue Data Quality.

type NumberOfWorkers:

integer

param NumberOfWorkers:

The number of G.1X workers to be used in the run. The default is 5.

type Timeout:

integer

param Timeout:

The timeout for a run in minutes. This is the maximum time that a run can consume resources before it is terminated and enters TIMEOUT status. The default is 2,880 minutes (48 hours).

type CreatedRulesetName:

string

param CreatedRulesetName:

A name for the ruleset.

type DataQualitySecurityConfiguration:

string

param DataQualitySecurityConfiguration:

The name of the security configuration created with the data quality encryption option.

type ClientToken:

string

param ClientToken:

Used for idempotency and is recommended to be set to a random ID (such as a UUID) to avoid creating or starting multiple instances of the same resource.

type AdditionalRunOptions:

dict

param AdditionalRunOptions:

Additional run options you can specify for a recommendation run.

CustomLogGroupPrefix (string) --

A custom prefix for the CloudWatch log group names. When specified, recommendation run logs are written to <CustomLogGroupPrefix>/error and <CustomLogGroupPrefix>/output instead of the default /aws-glue/data-quality/error and /aws-glue/data-quality/output log groups.

type RecommendationMode:

string

param RecommendationMode:

The mode that Glue Data Quality uses to recommend rules.

The default is BASIC.

rtype:

dict

returns:

Response Syntax

{
    'RunId': 'string'
}

Response Structure

(dict) --
- RunId (string) --
  
  The unique run identifier associated with this run.