boto3_datapipeline

Connection module for Amazon Data Pipeline using boto3.

Renamed from boto_datapipeline to boto3_datapipeline and rewritten to use the boto3 datapipeline client APIs directly via saltext.boto3.utils.boto3mod. The legacy boto2 code path (object-style access, retry loops) has been removed.

depends:

  • boto3 >= 1.28.0

  • botocore >= 1.31.0

configuration:

This module accepts explicit Data Pipeline credentials but can also utilize IAM roles assigned to the instance through Instance Profiles. Dynamic credentials are then automatically obtained from AWS API and no further configuration is necessary. More Information available at:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

If IAM roles are not used you need to specify them either in the minion’s config file or as a profile. For example, to specify them in the minion’s config file:

datapipeline.keyid: GKTADJGHEIQSXMKKRBJ08H
datapipeline.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs

A region may also be specified in the configuration:

datapipeline.region: us-east-1

It’s also possible to specify key, keyid and region via a profile, either as a passed in dict, or as a string to pull from pillars or minion config:

myprofile:
    keyid: GKTADJGHEIQSXMKKRBJ08H
    key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
    region: us-east-1

Added in version 1.0.0.

saltext.boto3.modules.boto3_datapipeline.__virtual__()[source]

Only load if boto3 is available.

saltext.boto3.modules.boto3_datapipeline.activate_pipeline(pipeline_id, region=None, key=None, keyid=None, profile=None)[source]

Start processing pipeline tasks. This function is idempotent.

pipeline_id (str):

The ID of the pipeline to activate.

region (str, optional):

The AWS region where the pipeline is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_datapipeline.activate_pipeline my_pipeline_id
saltext.boto3.modules.boto3_datapipeline.create_pipeline(name, unique_id, description='', region=None, key=None, keyid=None, profile=None)[source]

Create a new, empty pipeline. This function is idempotent.

name (str):

The name of the pipeline to create.

unique_id (str):

A unique identifier for the pipeline.

description (str, optional):

A description of the pipeline.

region (str, optional):

The AWS region where the pipeline will be created.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_datapipeline.create_pipeline my_name my_unique_id
saltext.boto3.modules.boto3_datapipeline.delete_pipeline(pipeline_id, region=None, key=None, keyid=None, profile=None)[source]

Delete a pipeline, its pipeline definition, and its run history. This function is idempotent.

pipeline_id (str):

The ID of the pipeline to delete.

region (str, optional):

The AWS region where the pipeline is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_datapipeline.delete_pipeline my_pipeline_id
saltext.boto3.modules.boto3_datapipeline.describe_pipelines(pipeline_ids, region=None, key=None, keyid=None, profile=None)[source]

Retrieve metadata about one or more pipelines.

pipeline_ids (list):

A list of pipeline IDs to retrieve metadata for.

region (str, optional):

The AWS region where the pipelines are located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_datapipeline.describe_pipelines ['my_pipeline_id']
saltext.boto3.modules.boto3_datapipeline.get_pipeline_definition(pipeline_id, version='latest', region=None, key=None, keyid=None, profile=None)[source]

Get the definition of the specified pipeline.

pipeline_id (str):

The ID of the pipeline to retrieve the definition for.

version (str, optional):

The version of the pipeline definition to retrieve. Defaults to “latest”.

region (str, optional):

The AWS region where the pipeline is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_datapipeline.get_pipeline_definition my_pipeline_id
saltext.boto3.modules.boto3_datapipeline.list_pipelines(region=None, key=None, keyid=None, profile=None)[source]

Get a list of pipeline ids and names for all pipelines.

region (str, optional):

The AWS region where the pipelines are located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_datapipeline.list_pipelines profile=myprofile
saltext.boto3.modules.boto3_datapipeline.pipeline_id_from_name(name, region=None, key=None, keyid=None, profile=None)[source]

Get the pipeline id, if it exists, for the given name.

name (str):

The name of the pipeline to look up.

region (str, optional):

The AWS region where the pipelines are located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_datapipeline.pipeline_id_from_name my_pipeline_name
saltext.boto3.modules.boto3_datapipeline.put_pipeline_definition(pipeline_id, pipeline_objects, parameter_objects=None, parameter_values=None, region=None, key=None, keyid=None, profile=None)[source]

Add tasks, schedules, and preconditions to the specified pipeline. This function is idempotent and will replace an existing definition.

pipeline_id (str):

The ID of the pipeline to update.

pipeline_objects (list):

A list of pipeline objects defining the tasks, schedules, and preconditions.

parameter_objects (list, optional):

A list of parameter objects for the pipeline.

parameter_values (list, optional):

A list of parameter values for the pipeline.

region (str, optional):

The AWS region where the pipeline is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_datapipeline.put_pipeline_definition my_pipeline_id my_pipeline_objects