2026/11/04 - Amazon S3 Tables - 6 new api methods
Changes Amazon S3 Tables now supports migrating AWS Glue Data Catalog tables into S3 Table buckets. This release adds the StartTableMigration, SyncTableMigration, UpdateTableMigration, CompleteTableMigration, GetTableMigration, and ListTableMigrations APIs.
Starts migrating an Amazon Web Services Glue Data Catalog table into a new table in a table bucket. The migration runs asynchronously: this operation returns the migration ID and its initial state, and you track progress with GetTableMigration.
The namespace must already exist, and a table with the specified name must not. The migration creates the table during planning and commits data to it as it runs. Until you call CompleteTableMigration, you can read the table but you can't write to it. If the migration fails or is cancelled, the table and the data already committed to it remain.
The migration first checks the source and the destination, and writes the table compatibility and object compatibility reports to the bucket that you specify in migrationOutput. If you set reviewMigrationPlan to true, the migration waits in the pendingReview state so that you can review the reports, and continues only when you call UpdateTableMigration with requestedState set to queued. If you don't continue or cancel the migration within 30 days, the migration fails. Otherwise it continues to migrate the data, and writes the object migration report when the migration ends. If planning fails, the migration writes no object migration report.
Permissions
You must have the s3tables:StartTableMigration permission to use this operation.
You must have the iam:PassRole permission on the role that you specify in roleARN, with a condition that allows passing it to migration.s3tables.amazonaws.com. The associated resource is the table bucket.
If destinationTable specifies SSE-KMS encryption, you must have the kms:DescribeKey and kms:GenerateDataKey permissions on the KMS key. You must also grant the S3 Tables maintenance principal access to the key. For more information, see Permissions requirements for S3 Tables SSE-KMS encryption.
See also: AWS API Documentation
Request Syntax
client.start_table_migration(
roleARN='string',
destinationTable={
'tableBucketARN': 'string',
'namespace': 'string',
'name': 'string',
'encryptionConfiguration': {
'sseAlgorithm': 'AES256'|'aws:kms',
'kmsKeyArn': 'string'
},
'storageClassConfiguration': {
'storageClass': 'STANDARD'|'INTELLIGENT_TIERING'
},
'tags': {
'string': 'string'
},
'properties': {
'writeFormat': 'parquet',
'icebergFormatVersion': 123,
'compressionCodec': 'zstd'|'snappy'|'gzip'|'uncompressed',
'dictionaryEncoding': True|False
},
'schema': {
'type': 'struct',
'fields': [
{
'id': 123,
'name': 'string',
'type': {...}|[...]|123|123.4|'string'|True|None,
'required': True|False,
'doc': 'string'
},
],
'schemaId': 123,
'identifierFieldIds': [
123,
]
},
'icebergSortOrder': {
'orderId': 123,
'fields': [
{
'sourceId': 123,
'transform': 'string',
'direction': 'asc'|'desc',
'nullOrder': 'nulls-first'|'nulls-last'
},
]
}
},
sourceType='glueTable',
source={
'glueTable': {
'catalogId': 'string',
'databaseName': 'string',
'tableName': 'string'
}
},
dataMigration={
'maintenanceSettings': {
'icebergCompaction': {
'targetFileSizeMB': 123,
'strategy': 'auto'|'binpack'|'sort'|'z-order'
},
'icebergSnapshotManagement': {
'minSnapshotsToKeep': 123,
'maxSnapshotAgeHours': 123
}
},
'migrationPolicies': {
'unmatchedSourceFieldsPolicy': 'fail'|'drop',
'incompatibleValuesPolicy': 'fail'|'setNull'
}
},
reviewMigrationPlan=True|False,
migrationOutput={
'bucket': 'string',
'prefixes': {
'tableCompatibilityReport': 'string',
'objectCompatibilityReport': 'string',
'objectMigrationReport': 'string'
}
}
)
string
[REQUIRED]
The Amazon Resource Name (ARN) of the IAM role that the migration uses to read the source table and its data and to write the destination table and the reports. The role must be in your account, and its trust policy must allow migration.s3tables.amazonaws.com to assume it. You can limit the trust policy to the destination table bucket with the aws:SourceArn and aws:SourceAccount condition keys.
dict
[REQUIRED]
The table to create and migrate the source into.
tableBucketARN (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the table bucket to create the table in.
namespace (string) -- [REQUIRED]
The namespace to create the table in.
name (string) -- [REQUIRED]
The name of the table to create.
encryptionConfiguration (dict) --
The encryption configuration for the table. If you don't specify one, the table uses the default encryption configuration of its table bucket. If you specify an encryption configuration, the migration role must have the s3tables:PutTableEncryption permission. If you choose SSE-KMS, the migration role must have the kms:DescribeKey, kms:GenerateDataKey and kms:Decrypt permissions on the KMS key, and the S3 Tables maintenance principal must have access to it. For more information, see Permissions requirements for S3 Tables SSE-KMS encryption.
sseAlgorithm (string) -- [REQUIRED]
The server-side encryption algorithm to use. Valid values are AES256 for S3-managed encryption keys, or aws:kms for Amazon Web Services KMS-managed encryption keys. If you choose SSE-KMS encryption you must grant the S3 Tables maintenance principal access to your KMS key. For more information, see Permissions requirements for S3 Tables SSE-KMS encryption.
kmsKeyArn (string) --
The Amazon Resource Name (ARN) of the KMS key to use for encryption. This field is required only when sseAlgorithm is set to aws:kms.
storageClassConfiguration (dict) --
The storage class configuration for the table. If you don't specify one, the table inherits the storage class configuration of its table bucket. If you specify a storage class configuration, the migration role must have the s3tables:PutTableStorageClass permission.
storageClass (string) -- [REQUIRED]
The storage class for the table or table bucket. Valid values include storage classes optimized for different access patterns and cost profiles.
tags (dict) --
The tags to add to the table. A tag is a key-value pair that you apply to your resources. For more information, see Tagging for cost allocation or attribute-based access control (ABAC). If you specify tags, the migration role must have the s3tables:TagResource permission.
(string) --
(string) --
properties (dict) --
The properties of the destination table, such as its Iceberg format version and the format of its data files.
writeFormat (string) --
The file format of the data files. The only supported format is parquet, the default.
icebergFormatVersion (integer) --
The Iceberg format version of the destination table. Valid values are 2 and 3. The default is 2.
compressionCodec (string) --
The compression codec of the data files. The default is zstd.
dictionaryEncoding (boolean) --
Specifies whether the data files use dictionary encoding. The default is true.
schema (dict) --
The schema of the table to create. If you don't specify a schema, the migration infers it from the source table, with every column optional. A schema that you specify must include every partition column of the source table. It can leave out other columns only if unmatchedSourceFieldsPolicy is drop. A struct column must have the same fields as the source column. Column default values aren't supported. If you mark a column as required, every source object must have a value for it: an object with a null value in a required column fails to migrate. The destination table is partitioned in the same way as the source table. To specify fields in icebergSortOrder, you must specify a schema.
type (string) -- [REQUIRED]
The type of the top-level schema, which is always a struct type as defined in the Apache Iceberg specification. This value must be struct.
fields (list) -- [REQUIRED]
The schema fields for the table. Each field defines a column in the table, including its name, type, and whether it is required.
(dict) --
Contains details about a schema field in the V2 format. This field format supports nested and complex data types such as struct, list, and map, in addition to primitive types.
id (integer) -- [REQUIRED]
The unique identifier for the schema field. Field IDs are used by Apache Iceberg to track schema evolution and maintain compatibility across schema changes.
name (string) -- [REQUIRED]
The name of the field.
type (:ref:`document<document>`) -- [REQUIRED]
The data type of the field. This can be a primitive type string such as boolean, int, long, float, double, string, binary, date, timestamp, or timestamptz, or a complex type represented as a JSON object for nested types such as struct, list, or map. For more information, see the Apache Iceberg schemas and data types documentation.
required (boolean) -- [REQUIRED]
A Boolean value that specifies whether values are required for each row in this field. If this is true, the field does not allow null values.
doc (string) --
An optional description of the field.
schemaId (integer) --
An optional unique identifier for the schema. Schema IDs are used by Apache Iceberg to track schema evolution.
identifierFieldIds (list) --
A list of field IDs that are used as the identifier fields for the table. Identifier fields uniquely identify a row in the table.
(integer) --
icebergSortOrder (dict) --
The sort order of the rows within each data file that the migration writes. A sort order can have up to five fields, each naming a column of the schema that you specify by its field ID, so to use a sort order you must specify a schema. StartTableMigration rejects a field that names no column of the schema, or whose transform doesn't fit the column's type.
orderId (integer) -- [REQUIRED]
The unique identifier for this sort order. If not specified, defaults to 1. The order ID is used by Apache Iceberg to track sort order evolution.
fields (list) -- [REQUIRED]
The list of sort fields that define how data is sorted within files. Each field specifies a source field, sort direction, and null ordering. This field is required if writeOrder is provided.
(dict) --
Defines a single sort field in an Iceberg sort order specification.
sourceId (integer) -- [REQUIRED]
The ID of the source schema field to sort by. This must reference a valid field ID from the table schema.
transform (string) -- [REQUIRED]
The transform to apply to the source field before sorting. Use identity to sort by the field value directly, or specify other transforms as needed.
direction (string) -- [REQUIRED]
The sort direction. Valid values are asc for ascending order or desc for descending order.
nullOrder (string) -- [REQUIRED]
Specifies how null values are ordered. Valid values are nulls-first to place nulls before non-null values, or nulls-last to place nulls after non-null values.
string
[REQUIRED]
The type of the source. It must match the member that you set in source.
dict
[REQUIRED]
The source table to migrate.
glueTable (dict) --
An Amazon Web Services Glue Data Catalog table to migrate. The table must be an external Apache Hive table whose data is stored as Parquet in a general purpose bucket in the same account and Region as the table bucket. Views, transactional (ACID) tables, tables in an open table format such as Apache Iceberg, and tables registered with Amazon Web Services Lake Formation can't be migrated. If the table can't be migrated, the migration fails, and the table compatibility report lists the reason.
catalogId (string) -- [REQUIRED]
The ID of the Data Catalog that contains the table. It must be your account ID.
databaseName (string) -- [REQUIRED]
The name of the database that contains the table.
tableName (string) -- [REQUIRED]
The name of the table.
dict
The maintenance settings and migration policies for the destination table.
maintenanceSettings (dict) --
The maintenance settings for the destination table. The migration applies them when it creates the table: snapshot management starts then, and compaction starts after you complete the migration. If you don't specify a setting, the migration uses its default: a target file size of 512 MB, the binpack compaction strategy, at least 3 snapshots, and a maximum snapshot age of 120 hours. Unreferenced file removal follows the table bucket's configuration after you complete the migration.
icebergCompaction (dict) --
The compaction settings.
targetFileSizeMB (integer) --
The target size, from 64 to 512 MB, of the data files that the migration writes, and of the files that compaction writes after you complete the migration. The default is 512.
strategy (string) --
The compaction strategy. The only valid value is binpack, which is the default.
icebergSnapshotManagement (dict) --
The snapshot management settings.
minSnapshotsToKeep (integer) --
The minimum number of snapshots to keep. The value must be 3 or more. The default is 3.
maxSnapshotAgeHours (integer) --
The maximum age, in hours, of a snapshot before it can be expired. The default is 120.
migrationPolicies (dict) --
The policies for source data that doesn't fit the destination schema.
unmatchedSourceFieldsPolicy (string) --
The migration policy for a source field that has no matching column in the destination schema. Valid values:
fail - Fails the migration. This is the default.
drop - Leaves source columns that the destination schema doesn't have out of the destination table, and doesn't migrate source objects outside the Amazon Web Services Glue table's partition locations.
A destination schema that leaves out a partition column or a field of a struct column fails the migration under either value.
incompatibleValuesPolicy (string) --
The migration policy for a source value that the destination column's type can't hold. Valid values:
fail - Fails the migration. This is the default.
setNull - Writes null for a value that the destination column's type can't hold. If a column's type in a source object is incompatible with the destination column's type, the column is null for that whole object.
A required column, or a type that migration doesn't support, fails the migration under either value.
boolean
Specifies whether the migration waits in the pendingReview state, so that you can review the compatibility reports before it migrates any data. If true, the migration continues when you call UpdateTableMigration with requestedState set to queued. If you don't continue or cancel the migration within 30 days, the migration fails. The default is false.
dict
[REQUIRED]
The bucket and prefixes that the migration writes its reports to. The bucket must be a general purpose bucket in your account and in the same Region as the table bucket, and the migration role must be able to write objects to it. Otherwise, the migration fails when it writes its first report.
bucket (string) -- [REQUIRED]
The general purpose bucket that the migration writes every report to.
prefixes (dict) --
The key prefix for each report.
tableCompatibilityReport (string) --
The key prefix for the table compatibility report, up to 512 bytes. If not specified, the report's keys start at the root of the bucket.
objectCompatibilityReport (string) --
The key prefix for the object compatibility report, up to 512 bytes. If not specified, the report's keys start at the root of the bucket.
objectMigrationReport (string) --
The key prefix for the object migration report, up to 512 bytes. If not specified, the report's keys start at the root of the bucket.
dict
Response Syntax
{
'migrationId': 'string',
'state': 'new'|'planning'|'pendingReview'|'queued'|'migrating'|'succeeded'|'failed'|'cancelling'|'cancelled'
}
Response Structure
(dict) --
migrationId (string) --
The ID of the migration.
state (string) --
The state of the migration when the request is accepted, such as planning.
Requests a state change for a migration. Identify the migration by its migrationId or by the tableARN of its destination table.
Set requestedState to queued to continue a migration in the pendingReview state; the migration moves to queued or migrating. Set it to cancelled to cancel a migration in the new, planning, pendingReview, queued or migrating state; the migration moves to cancelling, finishes the work in progress, and then ends in the cancelled state, or in the failed state if any objects failed to migrate. Data that the migration already committed stays in the destination table.
Permissions
You must have the s3tables:UpdateTableMigration permission to use this operation.
See also: AWS API Documentation
Request Syntax
client.update_table_migration(
tableARN='string',
migrationId='string',
requestedState='queued'|'cancelled',
updateDescription='string'
)
string
The Amazon Resource Name (ARN) of the destination table. Updates the table's most recent migration. Until the migration creates the destination table, specify migrationId. Specify either tableARN or migrationId, not both.
string
The ID of the migration to update. Specify either migrationId or tableARN, not both.
string
[REQUIRED]
The state to move the migration to: queued to continue a migration in the pendingReview state, or cancelled to cancel a migration in the new, planning, pendingReview, queued or migrating state.
string
A description of the update, up to 1,024 characters, with no control characters. GetTableMigration doesn't return it.
dict
Response Syntax
{
'migrationId': 'string',
'state': 'new'|'planning'|'pendingReview'|'queued'|'migrating'|'succeeded'|'failed'|'cancelling'|'cancelled',
'tableARN': 'string'
}
Response Structure
(dict) --
migrationId (string) --
The ID of the migration.
state (string) --
The state of the migration after the update, such as queued, migrating or cancelling.
tableARN (string) --
The Amazon Resource Name (ARN) of the destination table.
Migrates the changes in the source table since the previous start or sync, such as added and deleted source objects, into the destination table. The sync runs asynchronously: this operation returns the migration ID of the sync and its initial state, and you track progress with GetTableMigration.
A sync uses the source, destination and data migration settings of the table's migration, and plans the work and writes its reports in the same way as StartTableMigration. You can sync after a start or sync that succeeded, failed or was cancelled, if an earlier start or sync of the table has a valid plan ( hasValidPlan). You can't sync a table while a start or sync is in progress for it, and you can sync a table until you call CompleteTableMigration for it. If a source object that an earlier migration migrated has since been overwritten, the sync fails.
Permissions
You must have the s3tables:SyncTableMigration permission to use this operation.
You must have the iam:PassRole permission on the role that you specify in roleARN, with a condition that allows passing it to migration.s3tables.amazonaws.com. The associated resource is the table.
See also: AWS API Documentation
Request Syntax
client.sync_table_migration(
roleARN='string',
tableARN='string',
reviewMigrationPlan=True|False,
migrationOutput={
'bucket': 'string',
'prefixes': {
'tableCompatibilityReport': 'string',
'objectCompatibilityReport': 'string',
'objectMigrationReport': 'string'
}
}
)
string
[REQUIRED]
The Amazon Resource Name (ARN) of the IAM role that the sync uses to read the source table and its data and to write the destination table and the reports. The role must be in your account, and its trust policy must allow migration.s3tables.amazonaws.com to assume it. You can limit the trust policy to the destination table bucket with the aws:SourceArn and aws:SourceAccount condition keys.
string
[REQUIRED]
The Amazon Resource Name (ARN) of the destination table to sync. The table must have a migration, and you must not have called CompleteTableMigration for it.
boolean
Specifies whether the sync waits in the pendingReview state, so that you can review the compatibility reports before it migrates any data. If true, the migration continues when you call UpdateTableMigration with requestedState set to queued. If you don't continue or cancel the migration within 30 days, the migration fails. The default is false.
dict
[REQUIRED]
The bucket and prefixes that the sync writes its reports to. The bucket must be a general purpose bucket in your account and in the same Region as the table bucket, and the migration role must be able to write objects to it. Otherwise, the migration fails when it writes its first report.
bucket (string) -- [REQUIRED]
The general purpose bucket that the migration writes every report to.
prefixes (dict) --
The key prefix for each report.
tableCompatibilityReport (string) --
The key prefix for the table compatibility report, up to 512 bytes. If not specified, the report's keys start at the root of the bucket.
objectCompatibilityReport (string) --
The key prefix for the object compatibility report, up to 512 bytes. If not specified, the report's keys start at the root of the bucket.
objectMigrationReport (string) --
The key prefix for the object migration report, up to 512 bytes. If not specified, the report's keys start at the root of the bucket.
dict
Response Syntax
{
'migrationId': 'string',
'state': 'new'|'planning'|'pendingReview'|'queued'|'migrating'|'succeeded'|'failed'|'cancelling'|'cancelled'
}
Response Structure
(dict) --
migrationId (string) --
The migration ID of the sync.
state (string) --
The state of the migration when the request is accepted, such as planning.
Gets the details of a migration, including its state, the reason it failed, its statistics and the location of its reports. Identify the migration by its migrationId, or specify the tableARN of a destination table to get that table's most recent migration.
Permissions
You must have the s3tables:GetTableMigration permission to use this operation.
See also: AWS API Documentation
Request Syntax
client.get_table_migration(
migrationId='string',
tableARN='string'
)
string
The ID of the migration to get. Specify either migrationId or tableARN, not both.
string
The Amazon Resource Name (ARN) of the destination table. Gets the table's most recent migration. Specify either tableARN or migrationId, not both.
dict
Response Syntax
{
'migrationId': 'string',
'tableBucketARN': 'string',
'namespace': 'string',
'tableARN': 'string',
'name': 'string',
'type': 'start'|'sync',
'state': 'new'|'planning'|'pendingReview'|'queued'|'migrating'|'succeeded'|'failed'|'cancelling'|'cancelled',
'roleARN': 'string',
'reviewMigrationPlan': True|False,
'hasValidPlan': True|False,
'createdAt': datetime(2015, 1, 1),
'stateUpdatedAt': datetime(2015, 1, 1),
'failureReason': {
'code': 'rolePermissions'|'sourcePermissions'|'destinationPermissions'|'resourceNotFound'|'sourceNotSupported'|'objectMigrationReport'|'objectCompatibilityReport'|'tableCompatibilityReport'|'internalError'|'reportBucketNotSupported',
'detail': 'string'
},
'planningStats': {
'objectsScanned': 123,
'objectsPlanned': 123,
'partitionCount': 123
},
'migrationStats': {
'objectsMigrated': 123,
'objectsFailedMigration': 123,
'objectsPendingMigration': 123
},
'sourceType': 'glueTable',
'source': {
'glueTable': {
'catalogId': 'string',
'databaseName': 'string',
'tableName': 'string'
}
},
'dataMigration': {
'maintenanceSettings': {
'icebergCompaction': {
'targetFileSizeMB': 123,
'strategy': 'auto'|'binpack'|'sort'|'z-order'
},
'icebergSnapshotManagement': {
'minSnapshotsToKeep': 123,
'maxSnapshotAgeHours': 123
}
},
'migrationPolicies': {
'unmatchedSourceFieldsPolicy': 'fail'|'drop',
'incompatibleValuesPolicy': 'fail'|'setNull'
}
},
'migrationOutput': {
'bucket': 'string',
'tableCompatibilityReport': {
'prefix': 'string',
'key': 'string',
'checksumCRC64NVME': 'string',
'invalidFindings': {
'string': 123
}
},
'objectCompatibilityReport': {
'prefix': 'string',
'key': 'string',
'checksumCRC64NVME': 'string',
'invalidFindings': {
'string': 123
}
},
'objectMigrationReport': {
'prefix': 'string',
'key': 'string',
'checksumCRC64NVME': 'string',
'invalidFindings': {
'string': 123
}
}
}
}
Response Structure
(dict) --
migrationId (string) --
The ID of the migration.
tableBucketARN (string) --
The Amazon Resource Name (ARN) of the table bucket that contains the destination table.
namespace (string) --
The namespace of the destination table.
tableARN (string) --
The Amazon Resource Name (ARN) of the destination table, once the migration creates it.
name (string) --
The name of the destination table.
type (string) --
The type of the migration. Valid values:
start - A migration that StartTableMigration started.
sync - A migration that SyncTableMigration started.
state (string) --
The state of the migration. Valid values:
new - S3 Tables accepted the request.
planning - The migration is checking the source table, its objects and the destination for compatibility, and writing the compatibility reports.
pendingReview - Planning finished, and the migration waits for you to continue or cancel it with UpdateTableMigration.
queued - The migration is waiting to start migrating data.
migrating - The migration is running the plan from the planning state: it copies data and commits it to the destination table. A migration reaches this state only if hasValidPlan is true.
succeeded - Every planned object migrated.
failed - The migration failed. Check failureReason for the cause.
cancelling - You cancelled the migration, and it is finishing the work in progress.
cancelled - The migration is cancelled.
A migration ends in the succeeded, failed or cancelled state. You can cancel it until it ends.
roleARN (string) --
The Amazon Resource Name (ARN) of the IAM role that the migration uses.
reviewMigrationPlan (boolean) --
A Boolean value that specifies whether the migration waits in the pendingReview state, so that you can review the compatibility reports before it migrates any data.
hasValidPlan (boolean) --
A Boolean value that specifies whether planning produced a plan that the migration can run. If planning hasn't finished, this value isn't returned.
createdAt (datetime) --
The date and time that the migration was created.
stateUpdatedAt (datetime) --
The date and time that the migration's state last changed.
failureReason (dict) --
The reason that the migration failed. The code names the cause, such as a migration role that S3 Tables can't assume or an Amazon Web Services Glue table that doesn't exist, or the report that lists the findings. This value is returned only when the migration is in the failed state.
code (string) --
The failure code. It names the cause or the report that lists the findings. Valid values:
rolePermissions - S3 Tables couldn't assume the migration role, or the role doesn't have a permission that the migration needs. detail is the role ARN.
sourcePermissions - The migration role couldn't read the Amazon Web Services Glue table or the source objects. detail names the table ( database.table) or the source bucket.
destinationPermissions - The migration role couldn't write to the table bucket or the report bucket. detail names the bucket.
resourceNotFound - A resource that you specified doesn't exist, such as the Amazon Web Services Glue table or the source bucket. detail names it.
sourceNotSupported - The source table can't be migrated, for example because it is empty, has too many objects, or has columns or partitions that can't be migrated. The table compatibility report lists the findings.
tableCompatibilityReport - The migration couldn't create the destination table, for example because a table with that name already exists or the KMS key can't be used. For a sync, the destination table changed after the previous migration. The table compatibility report lists the finding.
objectCompatibilityReport - Some source objects can't be migrated, for example because they are archived, or because they were overwritten after an earlier migration migrated them. The object compatibility report lists them.
objectMigrationReport - Some source objects failed to migrate. The object migration report lists them and the cause for each.
reportBucketNotSupported - The report bucket can't receive the migration's reports, for example because it is in a different Region than the table bucket.
internalError - S3 Tables encountered an internal error. Retry the migration, and contact Amazon Web Services Support if the error persists.
detail (string) --
The resource that the failure applies to, such as the role ARN, the Amazon Web Services Glue table ( database.table) or a bucket. For sourceNotSupported, the reason that the source isn't supported, such as EMPTY_TABLE. This value isn't returned for the report codes or for internalError.
planningStats (dict) --
Statistics about the source and the partitions in the migration. This value is returned after planning finishes.
objectsScanned (integer) --
The number of source objects that the migration found under the source table's location, whether or not it migrates them. The object compatibility report has one row for each.
objectsPlanned (integer) --
The number of source objects in the migration. For a start, these are the data files in the Amazon Web Services Glue table's partitions, or under the table's location if the table isn't partitioned. For a sync, these are all the data files of the table after the sync, including files that earlier migrations migrated.
partitionCount (integer) --
The number of partitions in the destination table after this migration, or 0 if the source table isn't partitioned. This value is returned when planning produces a valid plan.
migrationStats (dict) --
The number of objects that migrated, failed or are pending. These counts are set when planning finishes and updated when the migration ends.
objectsMigrated (integer) --
The number of source objects whose data the migration wrote to the destination table. Migrated data can be queried after the migration commits it. If the migration fails or is cancelled, some migrated objects might not be committed.
objectsFailedMigration (integer) --
The number of source objects that didn't migrate, because they failed or because another object, such as an object in the same partition, failed. The object migration report lists the cause for each object.
objectsPendingMigration (integer) --
The number of source objects that haven't migrated or failed yet. When a migration fails or is cancelled, these are the objects that it didn't migrate.
sourceType (string) --
The type of the migration's source.
source (dict) --
The source table of the migration.
glueTable (dict) --
An Amazon Web Services Glue Data Catalog table to migrate. The table must be an external Apache Hive table whose data is stored as Parquet in a general purpose bucket in the same account and Region as the table bucket. Views, transactional (ACID) tables, tables in an open table format such as Apache Iceberg, and tables registered with Amazon Web Services Lake Formation can't be migrated. If the table can't be migrated, the migration fails, and the table compatibility report lists the reason.
catalogId (string) --
The ID of the Data Catalog that contains the table. It must be your account ID.
databaseName (string) --
The name of the database that contains the table.
tableName (string) --
The name of the table.
dataMigration (dict) --
The maintenance settings and migration policies for the destination table.
maintenanceSettings (dict) --
The maintenance settings for the destination table. The migration applies them when it creates the table: snapshot management starts then, and compaction starts after you complete the migration. If you don't specify a setting, the migration uses its default: a target file size of 512 MB, the binpack compaction strategy, at least 3 snapshots, and a maximum snapshot age of 120 hours. Unreferenced file removal follows the table bucket's configuration after you complete the migration.
icebergCompaction (dict) --
The compaction settings.
targetFileSizeMB (integer) --
The target size, from 64 to 512 MB, of the data files that the migration writes, and of the files that compaction writes after you complete the migration. The default is 512.
strategy (string) --
The compaction strategy. The only valid value is binpack, which is the default.
icebergSnapshotManagement (dict) --
The snapshot management settings.
minSnapshotsToKeep (integer) --
The minimum number of snapshots to keep. The value must be 3 or more. The default is 3.
maxSnapshotAgeHours (integer) --
The maximum age, in hours, of a snapshot before it can be expired. The default is 120.
migrationPolicies (dict) --
The policies for source data that doesn't fit the destination schema.
unmatchedSourceFieldsPolicy (string) --
The migration policy for a source field that has no matching column in the destination schema. Valid values:
fail - Fails the migration. This is the default.
drop - Leaves source columns that the destination schema doesn't have out of the destination table, and doesn't migrate source objects outside the Amazon Web Services Glue table's partition locations.
A destination schema that leaves out a partition column or a field of a struct column fails the migration under either value.
incompatibleValuesPolicy (string) --
The migration policy for a source value that the destination column's type can't hold. Valid values:
fail - Fails the migration. This is the default.
setNull - Writes null for a value that the destination column's type can't hold. If a column's type in a source object is incompatible with the destination column's type, the column is null for that whole object.
A required column, or a type that migration doesn't support, fails the migration under either value.
migrationOutput (dict) --
The bucket and prefixes for the migration's reports, and the location of each report once it is written.
bucket (string) --
The general purpose bucket that the migration writes every report to.
tableCompatibilityReport (dict) --
The table compatibility report: the sample object that the schema is inferred from, the table-level findings, and one row for each destination column with its type change and how the migration policies apply to it. The migration writes it when planning finishes, before any data migrates. If planning fails, the report might not be written.
prefix (string) --
The key prefix that the report is written under.
key (string) --
The key of the report object in the bucket. For the object compatibility and object migration reports, this is the report's manifest; for the table compatibility report, it is the report itself. This value is returned after the migration writes the report. A migration doesn't always write every report. For example, a migration that fails during planning doesn't write the object migration report.
checksumCRC64NVME (string) --
The Base64-encoded CRC64NVME checksum of the object that key names. For a report that has a manifest, the manifest lists the checksum of each report file. This value is returned after the migration writes the report.
invalidFindings (dict) --
For the compatibility reports, the number of blocking findings in each category. The object compatibility report counts each blocked object once, under its most severe finding. Categories with no blocking findings are left out.
(string) --
The category of a compatibility report finding, by what you change to fix it. Valid values:
access - A permission or a request setting. You don't need to change the source.
metadata - The Amazon Web Services Glue table definition, or properties of a source object, such as its size, storage class, encryption or Parquet types.
data - The contents of a source object, such as a partition value that can't be read, an object overwritten after an earlier migration, or a corrupt file.
internal - An internal error in S3 Tables. Retry the migration, and contact Amazon Web Services Support if the error persists.
(integer) --
objectCompatibilityReport (dict) --
The object compatibility report: one row for each source object under the source table's location, with whether the migration includes the object and its most severe finding. The migration writes it when planning finishes. If planning fails, the report might not be written.
prefix (string) --
The key prefix that the report is written under.
key (string) --
The key of the report object in the bucket. For the object compatibility and object migration reports, this is the report's manifest; for the table compatibility report, it is the report itself. This value is returned after the migration writes the report. A migration doesn't always write every report. For example, a migration that fails during planning doesn't write the object migration report.
checksumCRC64NVME (string) --
The Base64-encoded CRC64NVME checksum of the object that key names. For a report that has a manifest, the manifest lists the checksum of each report file. This value is returned after the migration writes the report.
invalidFindings (dict) --
For the compatibility reports, the number of blocking findings in each category. The object compatibility report counts each blocked object once, under its most severe finding. Categories with no blocking findings are left out.
(string) --
The category of a compatibility report finding, by what you change to fix it. Valid values:
access - A permission or a request setting. You don't need to change the source.
metadata - The Amazon Web Services Glue table definition, or properties of a source object, such as its size, storage class, encryption or Parquet types.
data - The contents of a source object, such as a partition value that can't be read, an object overwritten after an earlier migration, or a corrupt file.
internal - An internal error in S3 Tables. Retry the migration, and contact Amazon Web Services Support if the error persists.
(integer) --
objectMigrationReport (dict) --
The object migration report: one row for each source object that the migration tried to migrate, with whether it migrated, how it was transformed, and why it failed. The migration writes it once, when the migration ends. It isn't written if planning fails or if a sync has nothing to migrate.
prefix (string) --
The key prefix that the report is written under.
key (string) --
The key of the report object in the bucket. For the object compatibility and object migration reports, this is the report's manifest; for the table compatibility report, it is the report itself. This value is returned after the migration writes the report. A migration doesn't always write every report. For example, a migration that fails during planning doesn't write the object migration report.
checksumCRC64NVME (string) --
The Base64-encoded CRC64NVME checksum of the object that key names. For a report that has a manifest, the manifest lists the checksum of each report file. This value is returned after the migration writes the report.
invalidFindings (dict) --
For the compatibility reports, the number of blocking findings in each category. The object compatibility report counts each blocked object once, under its most severe finding. Categories with no blocking findings are left out.
(string) --
The category of a compatibility report finding, by what you change to fix it. Valid values:
access - A permission or a request setting. You don't need to change the source.
metadata - The Amazon Web Services Glue table definition, or properties of a source object, such as its size, storage class, encryption or Parquet types.
data - The contents of a source object, such as a partition value that can't be read, an object overwritten after an earlier migration, or a corrupt file.
internal - An internal error in S3 Tables. Retry the migration, and contact Amazon Web Services Support if the error persists.
(integer) --
Completes the migration of a table. The table's most recent migration must be in the succeeded, failed or cancelled state. After you complete the migration, you can't sync the table again, and you can write to and maintain the table as you do any other table: compaction starts, and unreferenced file removal follows the table bucket's configuration. You can't undo this operation. Calling it again for the same table returns a ConflictException.
Permissions
You must have the s3tables:CompleteTableMigration permission to use this operation.
See also: AWS API Documentation
Request Syntax
client.complete_table_migration(
tableARN='string'
)
string
[REQUIRED]
The Amazon Resource Name (ARN) of the migrated table to complete the migration of.
None
Lists the migrations whose destination is in a table bucket, a namespace or a table. Specify a tableBucketARN to list the migrations into a table bucket, and add a namespace, or a namespace and a name, to narrow the list. Or specify a tableARN to list the migrations into a table.
Migrations are listed newest first. Results are paginated: use maxResults to set the page size, and pass the nextToken from a response to retrieve the next page of results. A page can contain fewer migrations than maxResults, or none, and still return a nextToken. Continue until a response has no nextToken.
Permissions
You must have the s3tables:ListTableMigrations permission to use this operation.
See also: AWS API Documentation
Request Syntax
client.list_table_migrations(
tableBucketARN='string',
namespace='string',
name='string',
tableARN='string',
state='new'|'planning'|'pendingReview'|'queued'|'migrating'|'succeeded'|'failed'|'cancelling'|'cancelled',
hasValidPlan=True|False,
maxResults=123,
nextToken='string'
)
string
The Amazon Resource Name (ARN) of the table bucket to list migrations for.
string
The namespace to list migrations for. Requires tableBucketARN.
string
The name of the table to list migrations for. Requires tableBucketARN and namespace.
string
The Amazon Resource Name (ARN) of the table to list migrations for. The list includes the migrations into this table. To also list migrations into an earlier table of the same name, specify tableBucketARN, namespace and name instead. Specify either tableARN or tableBucketARN, not both.
string
The state to filter the list by. For the valid values, see the state member of GetTableMigration.
boolean
The hasValidPlan value to filter the list by. Migrations that are still planning aren't listed, whichever value you specify.
integer
The maximum number of migrations to return, from 1 to 1,000. If you don't specify a value, the default is 100.
string
The nextToken from a previous response, to get the next page of results.
dict
Response Syntax
{
'migrations': [
{
'migrationId': 'string',
'tableBucketARN': 'string',
'namespace': 'string',
'tableARN': 'string',
'name': 'string',
'type': 'start'|'sync',
'state': 'new'|'planning'|'pendingReview'|'queued'|'migrating'|'succeeded'|'failed'|'cancelling'|'cancelled',
'hasValidPlan': True|False,
'createdAt': datetime(2015, 1, 1),
'stateUpdatedAt': datetime(2015, 1, 1)
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
migrations (list) --
The migrations, newest first.
(dict) --
A summary of a migration.
migrationId (string) --
The ID of the migration.
tableBucketARN (string) --
The Amazon Resource Name (ARN) of the table bucket that contains the destination table.
namespace (string) --
The namespace of the destination table.
tableARN (string) --
The Amazon Resource Name (ARN) of the destination table, once the migration creates it.
name (string) --
The name of the destination table.
type (string) --
The type of the migration. Valid values:
start - A migration that StartTableMigration started.
sync - A migration that SyncTableMigration started.
state (string) --
The state of the migration. Valid values:
new - S3 Tables accepted the request.
planning - The migration is checking the source table, its objects and the destination for compatibility, and writing the compatibility reports.
pendingReview - Planning finished, and the migration waits for you to continue or cancel it with UpdateTableMigration.
queued - The migration is waiting to start migrating data.
migrating - The migration is running the plan from the planning state: it copies data and commits it to the destination table. A migration reaches this state only if hasValidPlan is true.
succeeded - Every planned object migrated.
failed - The migration failed. Check failureReason for the cause.
cancelling - You cancelled the migration, and it is finishing the work in progress.
cancelled - The migration is cancelled.
hasValidPlan (boolean) --
A Boolean value that specifies whether planning produced a plan that the migration can run. If planning hasn't finished, this value isn't returned.
createdAt (datetime) --
The date and time that the migration was created.
stateUpdatedAt (datetime) --
The date and time that the migration's state last changed.
nextToken (string) --
A token to pass as nextToken to get the next page of results. If there are no more results, this value isn't returned.