boto3_kinesis

Connection module for Amazon Kinesis using boto3.

Renamed from boto_kinesis to boto3_kinesis and rewritten to use the boto3 kinesis client APIs directly via saltext.boto3.utils.boto3mod. The legacy boto2 code path (object-style access, retry loops) has been removed.

depends:

• boto3 >= 1.28.0

• botocore >= 1.31.0

configuration:

This module accepts explicit kinesis credentials but can also utilize IAM roles assigned to the instance through Instance Profiles. Dynamic credentials are then automatically obtained from AWS API and no further configuration is necessary. More Information available at:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

If IAM roles are not used you need to specify them either in the minion’s config file or as a profile. For example, to specify them in the minion’s config file:

kinesis.keyid: GKTADJGHEIQSXMKKRBJ08H
kinesis.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs

A region may also be specified in the configuration:

kinesis.region: us-east-1

It’s also possible to specify key, keyid and region via a profile, either as a passed in dict, or as a string to pull from pillars or minion config:

myprofile:
    keyid: GKTADJGHEIQSXMKKRBJ08H
    key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
    region: us-east-1

Added in version 1.0.0.

saltext.boto3.modules.boto3_kinesis.__virtual__()

Only load if boto3 is available.

saltext.boto3.modules.boto3_kinesis.get_stream_when_active(stream_name, region=None, key=None, keyid=None, profile=None)

Get complete stream info from AWS, returning only when the stream is in the ACTIVE state. Continues to retry when stream is updating or creating. If the stream is deleted during retries, the loop will catch the error and break.

stream_name (str):

The name of the Kinesis stream to retrieve complete information for.

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.get_stream_when_active my_stream region=us-east-1

saltext.boto3.modules.boto3_kinesis.exists(stream_name, region=None, key=None, keyid=None, profile=None)

Check if the stream exists. Returns False and the error if it does not.

stream_name (str):

The name of the Kinesis stream to check for existence.

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.exists my_stream region=us-east-1

saltext.boto3.modules.boto3_kinesis.create_stream(stream_name, num_shards, region=None, key=None, keyid=None, profile=None)

Create a stream with name stream_name and initial number of shards num_shards.

stream_name (str):

The name of the Kinesis stream to create.

num_shards (int):

The initial number of shards for the stream.

region (str, optional):

The AWS region where the Kinesis stream will be created.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.create_stream my_stream N region=us-east-1

saltext.boto3.modules.boto3_kinesis.delete_stream(stream_name, region=None, key=None, keyid=None, profile=None)

Delete the stream with name stream_name. This cannot be undone! All data will be lost!!

stream_name (str):

The name of the Kinesis stream to delete.

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.delete_stream my_stream region=us-east-1

saltext.boto3.modules.boto3_kinesis.increase_stream_retention_period(stream_name, retention_hours, region=None, key=None, keyid=None, profile=None)

Increase stream retention period to retention_hours

stream_name (str):

The name of the Kinesis stream for which to increase the retention period.

retention_hours (int):

The new retention period for the stream, in hours.

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.increase_stream_retention_period my_stream N region=us-east-1

saltext.boto3.modules.boto3_kinesis.decrease_stream_retention_period(stream_name, retention_hours, region=None, key=None, keyid=None, profile=None)

Decrease stream retention period to retention_hours

stream_name (str):

The name of the Kinesis stream for which to decrease the retention period.

retention_hours (int):

The new retention period for the stream, in hours.

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.decrease_stream_retention_period my_stream N region=us-east-1

saltext.boto3.modules.boto3_kinesis.enable_enhanced_monitoring(stream_name, metrics, region=None, key=None, keyid=None, profile=None)

Enable enhanced monitoring for the specified shard-level metrics on stream stream_name

stream_name (str):

The name of the Kinesis stream for which to enable enhanced monitoring.

metrics (list):

A list of shard-level metrics to enable for enhanced monitoring.

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.enable_enhanced_monitoring my_stream ["metrics", "to", "enable"] region=us-east-1

saltext.boto3.modules.boto3_kinesis.disable_enhanced_monitoring(stream_name, metrics, region=None, key=None, keyid=None, profile=None)

Disable enhanced monitoring for the specified shard-level metrics on stream stream_name

stream_name (str):

The name of the Kinesis stream for which to disable enhanced monitoring.

metrics (list):

A list of shard-level metrics to disable for enhanced monitoring.

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.disable_enhanced_monitoring my_stream ["metrics", "to", "disable"] region=us-east-1

saltext.boto3.modules.boto3_kinesis.get_info_for_reshard(stream_details)

Collect some data: number of open shards, key range, etc. Modifies stream_details to add a sorted list of OpenShards. Returns (min_hash_key, max_hash_key, stream_details)

stream_details (dict):

The details of the Kinesis stream, including shard information. This dictionary will be modified to include a sorted list of open shards under the key “OpenShards”.

CLI Example:

salt myminion boto3_kinesis.get_info_for_reshard existing_stream_details

saltext.boto3.modules.boto3_kinesis.long_int(hash_key)

The hash key is a 128-bit int, sent as a string. It’s necessary to convert to int/long for comparison operations. This helper method handles python 2/3 incompatibility

hash_key (str):

The hash key as a string, representing a 128-bit integer.

CLI Example:

salt myminion boto3_kinesis.long_int some_MD5_hash_as_string

Returns:

long object if python 2.X, int object if python 3.X

saltext.boto3.modules.boto3_kinesis.reshard(stream_name, desired_size, force=False, region=None, key=None, keyid=None, profile=None)

Reshard a kinesis stream. Each call to this function will wait until the stream is ACTIVE, then make a single split or merge operation. This function decides where to split or merge with the assumption that the ultimate goal is a balanced partition space.

For safety, user must pass in force=True; otherwise, the function will dry run.

stream_name (str):

The name of the Kinesis stream to reshard.

desired_size (int):

The desired number of shards for the stream.

force (bool, optional):

Whether to force the reshard operation. Defaults to False.

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.reshard my_stream N True region=us-east-1

Returns:

True if a split or merge was found/performed, False if nothing is needed

saltext.boto3.modules.boto3_kinesis.list_streams(region=None, key=None, keyid=None, profile=None)

Return a list of all streams visible to the current account

region (str, optional):

The AWS region where the Kinesis stream is located.

key (str, optional):

The AWS secret access key.

keyid (str, optional):

The AWS access key ID.

profile (str, optional):

The profile to use for AWS credentials.

CLI Example:

salt myminion boto3_kinesis.list_streams