Amazon OpenSearch Ingestion

2023/04/26 - Amazon OpenSearch Ingestion - 14 new api methods

Changes  Initial release for OpenSearch Ingestion

GetPipeline (new) Link ¶

Retrieves information about an OpenSearch Ingestion pipeline.

See also: AWS API Documentation

Request Syntax

client.get_pipeline(
    PipelineName='string'
)
type PipelineName

string

param PipelineName

[REQUIRED]

The name of the pipeline to get information about.

rtype

dict

returns

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ]
                }
            },
        ]
    }
}

Response Structure

  • (dict) --

    • Pipeline (dict) --

      Detailed information about the requested pipeline.

      • PipelineName (string) --

        The name of the pipeline.

      • PipelineArn (string) --

        The Amazon Resource Name (ARN) of the pipeline.

      • MinUnits (integer) --

        The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

      • MaxUnits (integer) --

        The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

      • Status (string) --

        The current status of the pipeline.

      • StatusReason (dict) --

        The reason for the current status of the pipeline.

        • Description (string) --

          A description of why a pipeline has a certain status.

      • PipelineConfigurationBody (string) --

        The Data Prepper pipeline configuration in YAML format.

      • CreatedAt (datetime) --

        The date and time when the pipeline was created.

      • LastUpdatedAt (datetime) --

        The date and time when the pipeline was last updated.

      • IngestEndpointUrls (list) --

        The ingestion endpoints for the pipeline, which you can send data to.

        • (string) --

      • LogPublishingOptions (dict) --

        Key-value pairs that represent log publishing settings.

        • IsLoggingEnabled (boolean) --

          Whether logs should be published.

        • CloudWatchLogDestination (dict) --

          The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true .

          • LogGroup (string) --

            The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline .

      • VpcEndpoints (list) --

        The VPC interface endpoints that have access to the pipeline.

        • (dict) --

          An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.

          • VpcEndpointId (string) --

            The unique identifier of the endpoint.

          • VpcId (string) --

            The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.

          • VpcOptions (dict) --

            Information about the VPC, including associated subnets and security groups.

            • SubnetIds (list) --

              A list of subnet IDs associated with the VPC endpoint.

              • (string) --

            • SecurityGroupIds (list) --

              A list of security groups associated with the VPC endpoint.

              • (string) --

StopPipeline (new) Link ¶

Stops an OpenSearch Ingestion pipeline. For more information, see Stopping pipelines .

See also: AWS API Documentation

Request Syntax

client.stop_pipeline(
    PipelineName='string'
)
type PipelineName

string

param PipelineName

[REQUIRED]

The name of the pipeline to stop.

rtype

dict

returns

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ]
                }
            },
        ]
    }
}

Response Structure

  • (dict) --

    • Pipeline (dict) --

      Information about an existing OpenSearch Ingestion pipeline.

      • PipelineName (string) --

        The name of the pipeline.

      • PipelineArn (string) --

        The Amazon Resource Name (ARN) of the pipeline.

      • MinUnits (integer) --

        The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

      • MaxUnits (integer) --

        The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

      • Status (string) --

        The current status of the pipeline.

      • StatusReason (dict) --

        The reason for the current status of the pipeline.

        • Description (string) --

          A description of why a pipeline has a certain status.

      • PipelineConfigurationBody (string) --

        The Data Prepper pipeline configuration in YAML format.

      • CreatedAt (datetime) --

        The date and time when the pipeline was created.

      • LastUpdatedAt (datetime) --

        The date and time when the pipeline was last updated.

      • IngestEndpointUrls (list) --

        The ingestion endpoints for the pipeline, which you can send data to.

        • (string) --

      • LogPublishingOptions (dict) --

        Key-value pairs that represent log publishing settings.

        • IsLoggingEnabled (boolean) --

          Whether logs should be published.

        • CloudWatchLogDestination (dict) --

          The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true .

          • LogGroup (string) --

            The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline .

      • VpcEndpoints (list) --

        The VPC interface endpoints that have access to the pipeline.

        • (dict) --

          An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.

          • VpcEndpointId (string) --

            The unique identifier of the endpoint.

          • VpcId (string) --

            The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.

          • VpcOptions (dict) --

            Information about the VPC, including associated subnets and security groups.

            • SubnetIds (list) --

              A list of subnet IDs associated with the VPC endpoint.

              • (string) --

            • SecurityGroupIds (list) --

              A list of security groups associated with the VPC endpoint.

              • (string) --

DeletePipeline (new) Link ¶

Deletes an OpenSearch Ingestion pipeline. For more information, see Deleting pipelines .

See also: AWS API Documentation

Request Syntax

client.delete_pipeline(
    PipelineName='string'
)
type PipelineName

string

param PipelineName

[REQUIRED]

The name of the pipeline to delete.

rtype

dict

returns

Response Syntax

{}

Response Structure

  • (dict) --

GetPipelineBlueprint (new) Link ¶

Retrieves information about a specific blueprint for OpenSearch Ingestion. Blueprints are templates for the configuration needed for a CreatePipeline request.

See also: AWS API Documentation

Request Syntax

client.get_pipeline_blueprint(
    BlueprintName='string'
)
type BlueprintName

string

param BlueprintName

[REQUIRED]

The name of the blueprint to retrieve.

rtype

dict

returns

Response Syntax

{
    'Blueprint': {
        'BlueprintName': 'string',
        'PipelineConfigurationBody': 'string'
    }
}

Response Structure

  • (dict) --

    • Blueprint (dict) --

      The requested blueprint in YAML format.

      • BlueprintName (string) --

        The name of the blueprint.

      • PipelineConfigurationBody (string) --

        The YAML configuration of the blueprint.

UpdatePipeline (new) Link ¶

Updates an OpenSearch Ingestion pipeline. For more information, see Creating and managing OpenSearch Ingestion pipelines .

See also: AWS API Documentation

Request Syntax

client.update_pipeline(
    PipelineName='string',
    MinUnits=123,
    MaxUnits=123,
    PipelineConfigurationBody='string',
    LogPublishingOptions={
        'IsLoggingEnabled': True|False,
        'CloudWatchLogDestination': {
            'LogGroup': 'string'
        }
    }
)
type PipelineName

string

param PipelineName

[REQUIRED]

The name of the pipeline to update.

type MinUnits

integer

param MinUnits

The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

type MaxUnits

integer

param MaxUnits

The maximum pipeline capacity, in Ingestion Compute Units (ICUs)

type PipelineConfigurationBody

string

param PipelineConfigurationBody

The pipeline configuration in YAML format. The command accepts the pipeline configuration as a string or within a .yaml file. If you provide the configuration as a string, each new line must be escaped with \n .

type LogPublishingOptions

dict

param LogPublishingOptions

Key-value pairs to configure log publishing.

  • IsLoggingEnabled (boolean) --

    Whether logs should be published.

  • CloudWatchLogDestination (dict) --

    The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true .

    • LogGroup (string) -- [REQUIRED]

      The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline .

rtype

dict

returns

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ]
                }
            },
        ]
    }
}

Response Structure

  • (dict) --

    • Pipeline (dict) --

      Container for information about the updated pipeline.

      • PipelineName (string) --

        The name of the pipeline.

      • PipelineArn (string) --

        The Amazon Resource Name (ARN) of the pipeline.

      • MinUnits (integer) --

        The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

      • MaxUnits (integer) --

        The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

      • Status (string) --

        The current status of the pipeline.

      • StatusReason (dict) --

        The reason for the current status of the pipeline.

        • Description (string) --

          A description of why a pipeline has a certain status.

      • PipelineConfigurationBody (string) --

        The Data Prepper pipeline configuration in YAML format.

      • CreatedAt (datetime) --

        The date and time when the pipeline was created.

      • LastUpdatedAt (datetime) --

        The date and time when the pipeline was last updated.

      • IngestEndpointUrls (list) --

        The ingestion endpoints for the pipeline, which you can send data to.

        • (string) --

      • LogPublishingOptions (dict) --

        Key-value pairs that represent log publishing settings.

        • IsLoggingEnabled (boolean) --

          Whether logs should be published.

        • CloudWatchLogDestination (dict) --

          The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true .

          • LogGroup (string) --

            The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline .

      • VpcEndpoints (list) --

        The VPC interface endpoints that have access to the pipeline.

        • (dict) --

          An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.

          • VpcEndpointId (string) --

            The unique identifier of the endpoint.

          • VpcId (string) --

            The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.

          • VpcOptions (dict) --

            Information about the VPC, including associated subnets and security groups.

            • SubnetIds (list) --

              A list of subnet IDs associated with the VPC endpoint.

              • (string) --

            • SecurityGroupIds (list) --

              A list of security groups associated with the VPC endpoint.

              • (string) --

GetPipelineChangeProgress (new) Link ¶

Returns progress information for the current change happening on an OpenSearch Ingestion pipeline. Currently, this operation only returns information when a pipeline is being created.

For more information, see Creating and managing OpenSearch Ingestion pipelines .

See also: AWS API Documentation

Request Syntax

client.get_pipeline_change_progress(
    PipelineName='string'
)
type PipelineName

string

param PipelineName

[REQUIRED]

The name of the pipeline.

rtype

dict

returns

Response Syntax

{
    'ChangeProgressStatuses': [
        {
            'StartTime': datetime(2015, 1, 1),
            'Status': 'PENDING'|'IN_PROGRESS'|'COMPLETED'|'FAILED',
            'TotalNumberOfStages': 123,
            'ChangeProgressStages': [
                {
                    'Name': 'string',
                    'Status': 'PENDING'|'IN_PROGRESS'|'COMPLETED'|'FAILED',
                    'Description': 'string',
                    'LastUpdatedAt': datetime(2015, 1, 1)
                },
            ]
        },
    ]
}

Response Structure

  • (dict) --

    • ChangeProgressStatuses (list) --

      The current status of the change happening on the pipeline.

      • (dict) --

        The progress details of a pipeline configuration change.

        • StartTime (datetime) --

          The time at which the configuration change is made on the pipeline.

        • Status (string) --

          The overall status of the pipeline configuration change.

        • TotalNumberOfStages (integer) --

          The total number of stages required for the pipeline configuration change.

        • ChangeProgressStages (list) --

          Information about the stages that the pipeline is going through to perform the configuration change.

          • (dict) --

            Progress details for a specific stage of a pipeline configuration change.

            • Name (string) --

              The name of the stage.

            • Status (string) --

              The current status of the stage that the change is in.

            • Description (string) --

              A description of the stage.

            • LastUpdatedAt (datetime) --

              The most recent updated timestamp of the stage.

ListPipelines (new) Link ¶

Lists all OpenSearch Ingestion pipelines in the current Amazon Web Services account and Region. For more information, see Creating and managing OpenSearch Ingestion pipelines .

See also: AWS API Documentation

Request Syntax

client.list_pipelines(
    MaxResults=123,
    NextToken='string'
)
type MaxResults

integer

param MaxResults

An optional parameter that specifies the maximum number of results to return. You can use nextToken to get the next page of results.

type NextToken

string

param NextToken

If your initial ListPipelines operation returns a nextToken , you can include the returned nextToken in subsequent ListPipelines operations, which returns results in the next page.

rtype

dict

returns

Response Syntax

{
    'NextToken': 'string',
    'Pipelines': [
        {
            'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
            'StatusReason': {
                'Description': 'string'
            },
            'PipelineName': 'string',
            'PipelineArn': 'string',
            'MinUnits': 123,
            'MaxUnits': 123,
            'CreatedAt': datetime(2015, 1, 1),
            'LastUpdatedAt': datetime(2015, 1, 1)
        },
    ]
}

Response Structure

  • (dict) --

    • NextToken (string) --

      When nextToken is returned, there are more results available. The value of nextToken is a unique pagination token for each page. Make the call again using the returned token to retrieve the next page.

    • Pipelines (list) --

      A list of all existing Data Prepper pipelines.

      • (dict) --

        Summary information for an OpenSearch Ingestion pipeline.

        • Status (string) --

          The current status of the pipeline.

        • StatusReason (dict) --

          Information about a pipeline's current status.

          • Description (string) --

            A description of why a pipeline has a certain status.

        • PipelineName (string) --

          The name of the pipeline.

        • PipelineArn (string) --

          The Amazon Resource Name (ARN) of the pipeline.

        • MinUnits (integer) --

          The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

        • MaxUnits (integer) --

          The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

        • CreatedAt (datetime) --

          The date and time when the pipeline was created.

        • LastUpdatedAt (datetime) --

          The date and time when the pipeline was last updated.

ListTagsForResource (new) Link ¶

Lists all resource tags associated with an OpenSearch Ingestion pipeline. For more information, see Tagging OpenSearch Ingestion pipelines .

See also: AWS API Documentation

Request Syntax

client.list_tags_for_resource(
    Arn='string'
)
type Arn

string

param Arn

[REQUIRED]

The Amazon Resource Name (ARN) of the pipeline to retrieve tags for.

rtype

dict

returns

Response Syntax

{
    'Tags': [
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
}

Response Structure

  • (dict) --

    • Tags (list) --

      A list of tags associated with the given pipeline.

      • (dict) --

        A tag (key-value pair) for an OpenSearch Ingestion pipeline.

        • Key (string) --

          The tag key. Tag keys must be unique for the pipeline to which they are attached.

        • Value (string) --

          The value assigned to the corresponding tag key. Tag values can be null and don't have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

CreatePipeline (new) Link ¶

Creates an OpenSearch Ingestion pipeline. For more information, see Creating and managing OpenSearch Ingestion pipelines .

See also: AWS API Documentation

Request Syntax

client.create_pipeline(
    PipelineName='string',
    MinUnits=123,
    MaxUnits=123,
    PipelineConfigurationBody='string',
    LogPublishingOptions={
        'IsLoggingEnabled': True|False,
        'CloudWatchLogDestination': {
            'LogGroup': 'string'
        }
    },
    VpcOptions={
        'SubnetIds': [
            'string',
        ],
        'SecurityGroupIds': [
            'string',
        ]
    },
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)
type PipelineName

string

param PipelineName

[REQUIRED]

The name of the OpenSearch Ingestion pipeline to create. Pipeline names are unique across the pipelines owned by an account within an Amazon Web Services Region.

type MinUnits

integer

param MinUnits

[REQUIRED]

The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

type MaxUnits

integer

param MaxUnits

[REQUIRED]

The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

type PipelineConfigurationBody

string

param PipelineConfigurationBody

[REQUIRED]

The pipeline configuration in YAML format. The command accepts the pipeline configuration as a string or within a .yaml file. If you provide the configuration as a string, each new line must be escaped with \n .

type LogPublishingOptions

dict

param LogPublishingOptions

Key-value pairs to configure log publishing.

  • IsLoggingEnabled (boolean) --

    Whether logs should be published.

  • CloudWatchLogDestination (dict) --

    The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true .

    • LogGroup (string) -- [REQUIRED]

      The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline .

type VpcOptions

dict

param VpcOptions

Container for the values required to configure VPC access for the pipeline. If you don't specify these values, OpenSearch Ingestion creates the pipeline with a public endpoint.

  • SubnetIds (list) -- [REQUIRED]

    A list of subnet IDs associated with the VPC endpoint.

    • (string) --

  • SecurityGroupIds (list) --

    A list of security groups associated with the VPC endpoint.

    • (string) --

type Tags

list

param Tags

List of tags to add to the pipeline upon creation.

  • (dict) --

    A tag (key-value pair) for an OpenSearch Ingestion pipeline.

    • Key (string) -- [REQUIRED]

      The tag key. Tag keys must be unique for the pipeline to which they are attached.

    • Value (string) -- [REQUIRED]

      The value assigned to the corresponding tag key. Tag values can be null and don't have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

rtype

dict

returns

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ]
                }
            },
        ]
    }
}

Response Structure

  • (dict) --

    • Pipeline (dict) --

      Container for information about the created pipeline.

      • PipelineName (string) --

        The name of the pipeline.

      • PipelineArn (string) --

        The Amazon Resource Name (ARN) of the pipeline.

      • MinUnits (integer) --

        The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

      • MaxUnits (integer) --

        The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

      • Status (string) --

        The current status of the pipeline.

      • StatusReason (dict) --

        The reason for the current status of the pipeline.

        • Description (string) --

          A description of why a pipeline has a certain status.

      • PipelineConfigurationBody (string) --

        The Data Prepper pipeline configuration in YAML format.

      • CreatedAt (datetime) --

        The date and time when the pipeline was created.

      • LastUpdatedAt (datetime) --

        The date and time when the pipeline was last updated.

      • IngestEndpointUrls (list) --

        The ingestion endpoints for the pipeline, which you can send data to.

        • (string) --

      • LogPublishingOptions (dict) --

        Key-value pairs that represent log publishing settings.

        • IsLoggingEnabled (boolean) --

          Whether logs should be published.

        • CloudWatchLogDestination (dict) --

          The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true .

          • LogGroup (string) --

            The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline .

      • VpcEndpoints (list) --

        The VPC interface endpoints that have access to the pipeline.

        • (dict) --

          An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.

          • VpcEndpointId (string) --

            The unique identifier of the endpoint.

          • VpcId (string) --

            The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.

          • VpcOptions (dict) --

            Information about the VPC, including associated subnets and security groups.

            • SubnetIds (list) --

              A list of subnet IDs associated with the VPC endpoint.

              • (string) --

            • SecurityGroupIds (list) --

              A list of security groups associated with the VPC endpoint.

              • (string) --

UntagResource (new) Link ¶

Removes one or more tags from an OpenSearch Ingestion pipeline. For more information, see Tagging OpenSearch Ingestion pipelines .

See also: AWS API Documentation

Request Syntax

client.untag_resource(
    Arn='string',
    TagKeys=[
        'string',
    ]
)
type Arn

string

param Arn

[REQUIRED]

The Amazon Resource Name (ARN) of the pipeline to remove tags from.

type TagKeys

list

param TagKeys

[REQUIRED]

The tag keys to remove.

  • (string) --

rtype

dict

returns

Response Syntax

{}

Response Structure

  • (dict) --

ListPipelineBlueprints (new) Link ¶

Retrieves a list of all available blueprints for Data Prepper.

See also: AWS API Documentation

Request Syntax

client.list_pipeline_blueprints()
rtype

dict

returns

Response Syntax

{
    'Blueprints': [
        {
            'BlueprintName': 'string'
        },
    ]
}

Response Structure

  • (dict) --

    • Blueprints (list) --

      A list of available blueprints for Data Prepper.

      • (dict) --

        A summary of an OpenSearch Ingestion blueprint.

        • BlueprintName (string) --

          The name of the blueprint.

StartPipeline (new) Link ¶

Starts an OpenSearch Ingestion pipeline. For more information, see Starting pipelines .

See also: AWS API Documentation

Request Syntax

client.start_pipeline(
    PipelineName='string'
)
type PipelineName

string

param PipelineName

[REQUIRED]

The name of the pipeline to start.

rtype

dict

returns

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ]
                }
            },
        ]
    }
}

Response Structure

  • (dict) --

    • Pipeline (dict) --

      Information about an existing OpenSearch Ingestion pipeline.

      • PipelineName (string) --

        The name of the pipeline.

      • PipelineArn (string) --

        The Amazon Resource Name (ARN) of the pipeline.

      • MinUnits (integer) --

        The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

      • MaxUnits (integer) --

        The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

      • Status (string) --

        The current status of the pipeline.

      • StatusReason (dict) --

        The reason for the current status of the pipeline.

        • Description (string) --

          A description of why a pipeline has a certain status.

      • PipelineConfigurationBody (string) --

        The Data Prepper pipeline configuration in YAML format.

      • CreatedAt (datetime) --

        The date and time when the pipeline was created.

      • LastUpdatedAt (datetime) --

        The date and time when the pipeline was last updated.

      • IngestEndpointUrls (list) --

        The ingestion endpoints for the pipeline, which you can send data to.

        • (string) --

      • LogPublishingOptions (dict) --

        Key-value pairs that represent log publishing settings.

        • IsLoggingEnabled (boolean) --

          Whether logs should be published.

        • CloudWatchLogDestination (dict) --

          The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true .

          • LogGroup (string) --

            The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline .

      • VpcEndpoints (list) --

        The VPC interface endpoints that have access to the pipeline.

        • (dict) --

          An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.

          • VpcEndpointId (string) --

            The unique identifier of the endpoint.

          • VpcId (string) --

            The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.

          • VpcOptions (dict) --

            Information about the VPC, including associated subnets and security groups.

            • SubnetIds (list) --

              A list of subnet IDs associated with the VPC endpoint.

              • (string) --

            • SecurityGroupIds (list) --

              A list of security groups associated with the VPC endpoint.

              • (string) --

ValidatePipeline (new) Link ¶

Checks whether an OpenSearch Ingestion pipeline configuration is valid prior to creation. For more information, see Creating and managing OpenSearch Ingestion pipelines .

See also: AWS API Documentation

Request Syntax

client.validate_pipeline(
    PipelineConfigurationBody='string'
)
type PipelineConfigurationBody

string

param PipelineConfigurationBody

[REQUIRED]

The pipeline configuration in YAML format. The command accepts the pipeline configuration as a string or within a .yaml file. If you provide the configuration as a string, each new line must be escaped with \n .

rtype

dict

returns

Response Syntax

{
    'isValid': True|False,
    'Errors': [
        {
            'Message': 'string'
        },
    ]
}

Response Structure

  • (dict) --

    • isValid (boolean) --

      A boolean indicating whether or not the pipeline configuration is valid.

    • Errors (list) --

      A list of errors if the configuration is invalid.

      • (dict) --

        A validation message associated with a ValidatePipeline request in OpenSearch Ingestion.

        • Message (string) --

          The validation message.

TagResource (new) Link ¶

Tags an OpenSearch Ingestion pipeline. For more information, see Tagging OpenSearch Ingestion pipelines .

See also: AWS API Documentation

Request Syntax

client.tag_resource(
    Arn='string',
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)
type Arn

string

param Arn

[REQUIRED]

The Amazon Resource Name (ARN) of the pipeline to tag.

type Tags

list

param Tags

[REQUIRED]

The list of key-value tags to add to the pipeline.

  • (dict) --

    A tag (key-value pair) for an OpenSearch Ingestion pipeline.

    • Key (string) -- [REQUIRED]

      The tag key. Tag keys must be unique for the pipeline to which they are attached.

    • Value (string) -- [REQUIRED]

      The value assigned to the corresponding tag key. Tag values can be null and don't have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

rtype

dict

returns

Response Syntax

{}

Response Structure

  • (dict) --