Automatic IAM User Access Key Rotation

The company I work at uses a hybrid cloud setup. While modern applications are able to utilize HashiCorp Vault to provision temporary access, applications that continue to run on legacy on premises servers are unable to do so. In these scenarios, an IAM user needs to be provisioned in order to generate AWS Access Key and Secret Access Key to grant programmatic access.

Following best security practices, these keys need to be rotated from time to time. Having many of these access keys generated would increase the team’s operational workload.

Here’s how I automated this process.

AWS Services Used

  • SNS Topic
  • Secrets Manager
  • EventBridge (Formerly known as CloudWatch Events)
  • AWS Lambda
  • Identity Access Management

Workflow

It’s important for us to get a clear picture of what we’re trying to achieve and how all of these AWS services are connected to one another. Here’s how it works

  • An IAM User is created with Access Key generated
  • 3 Event Bridge rules are used to trigger a lambda function based on number of days elapsed.
    • Every 90 days – Create new access keys
    • Every 104 days – Deactivate the old access key
    • Every 128 days – Delete the old access key
  • Lambda function receives 1) IAM Username 2) The action to perform and performs the 3 actions above accordingly on IAM User.
  • Secrets Manager stores the new access key and holds records of previous access keys
Create Access Key
Deactivate Access Key
Delete Access Key

Steps

  1. Create the IAM user and generate AWS Access Key.
Add user with “Programmatic access” type selected
Access Key Generated
  • For all IAM user and roles created, skip attaching or creating IAM policies. This will be revisited once we have created all necessary resources.

2. Create IAM Role for our Lambda function.

  • When creating this role, set the trusted entity type to be AWS service and the usecase to be “Lambda”.

3. Create Secrets Manager Secret with the secret name matching the name of the IAM username you intended.

Add new secret
  • Select “Other type of secrets” option as secret type
  • For secret key names, use “AccessKey” for IAM Access Key and “SecretKey” for IAM secret access key
  • Keep key rotation disabled

4. Create Lambda Function that will process IAM key rotation requests.

Use the following code for the lambda function. Be sure to modify parts of the code that have been surrounded by ankle brackets.

import json
import boto3
import base64
import datetime
import os
from datetime import date
from botocore.exceptions import ClientError
AWS_REGION_NAME = '<your region here>'
SNS_TOPIC_ARN = '<your sns topic arn>'
ACCESS_KEY_SECRET_NAME = '<iam username>'
iam = boto3.client('iam')
secretmanager = boto3.client('secretsmanager')
sns = boto3.client('sns', region_name=<AWS_REGION_NAME)

def create_key(iam_username):
    '''Generates a new access key on behalf of the user and stores the new
    access key in secrets manager. Then, send a notification email to users to
    notify them to rotate the key for their applications. It returns
    a JSON with status 200 if successful and 500 if error occurs.

    Arguments:
    iam_username - The iam user's username as a string.
    '''

    try:
        response = iam.create_access_key(UserName=iam_username)
        access_key = response['AccessKey']['AccessKeyId']
        secret_key = response['AccessKey']['SecretAccessKey']
        json_data = json.dumps(
            {'AccessKey': access_key, 'SecretKey': secret_key})
        secretmanager.put_secret_value(
            SecretId=iam_username, SecretString=json_data)

        iam_user_details = get_iam_user_details(iam_username)

        emailmsg = 'Hello,\n\n' \
            'A new access key has been created for key rotation. \n\n' \
            f'Access Key Id: {access_key}\n' \
            f'Secrets Manager Secret Id: {iam_username}'

        emailmsg = f'{emailmsg}\n\n' \
            f'Please obtain the new access key information from ' \
            'secrets manager using the secret Id provided above in ' \
            f'{AWS_REGION_NAME} and update your application within 14 days ' \
            'to avoid interruption.\n'

        sns.publish(TopicArn=SNS_TOPIC_ARN, Message=emailmsg,
                    Subject=f'AWS Access Key Rotation: New key is available for '
                            f'{iam_username}')
        print(f'New access key has been created for {iam_username}')
        return {'status': 200}
    except ClientError as e:
        print(e)
        return {"status": 500}


def deactive_key(iam_username):
    '''Finds the secret that stores the user's previous access key
    and mark it as inactive. Then, send a notification email to users to remind
    them to rotate the key for their applications. It returns
    a JSON with status 200 if successful and 500 if error occurs.

    Arguments:
    iam_username - The iam user's username as a string.
    '''

    try:
        previous_secret_value = secretmanager.get_secret_value(
            SecretId=iam_username, VersionStage='AWSPREVIOUS')
        previous_secret_data = json.loads(
            previous_secret_value['SecretString'])
        previous_access_key = previous_secret_data['AccessKey']

        iam_user_details = get_iam_user_details(iam_username)

        print(
            f'deactivating access key {previous_access_key} '
            f'for IAM user {iam_username}')

        iam.update_access_key(AccessKeyId=previous_access_key,
                              Status='Inactive', UserName=iam_username)

        emailmsg = f'Hello,\n\n' \
            f'The previous access key {previous_access_key}'

        emailmsg = f'{emailmsg} has been disabled for {iam_username}.\n\n' \
            f'This key will be deleted in the next 14 days. ' \
            f'If your application has lost access, be sure to update the ' \
            f'access key.\n You can find the new key by looking up the secret ' \
            f'"{iam_username}" under secrets manager via AWS Console ' \
            f'in {AWS_REGION_NAME}.\n\n'
 
        sns.publish(
            TopicArn=SNS_TOPIC_ARN, Message=emailmsg,
            Subject='AWS Access Key Rotation: Previous key deactivated for '
                    f'{iam_username}')
        print('Access Key has been deacivated')
        return {'status': 200}
    except ClientError as e:
        print(e)
        return {'status': 500}


def delete_key(iam_username):
    '''Deletes the deactivated access key in the given iam user. Returns
    a JSON with status 200 if successful, 500 for error and 400 for
    if secrets don't match

    Arguments:
    iam_username - The iam user's username as a string.
    '''
    try:
        previous_secret_value = secretmanager.get_secret_value(
            SecretId=iam_username, VersionStage='AWSPREVIOUS')
        previous_secret_string = json.loads(
            previous_secret_value['SecretString'])
        previous_access_key_id = previous_secret_string['AccessKey']
        pprint(f'previous_access_key_id: {previous_access_key_id}')
        keylist = iam.list_access_keys(UserName=iam_username)[
            'AccessKeyMetadata']

        for key in keylist:
            key_status = key['Status']
            key_id = key['AccessKeyId']

            print(f'key id: {key_id}')
            print(f'key status: {key_status}')

            if key_status == "Inactive":
                if previous_access_key_id == key_id:
                    print('Deleting previous access key from IAM user')
                    iam.delete_access_key(
                        UserName=iam_username, AccessKeyId=key_id)
                    print(f'Previous access key: '
                          f'{key_id} has been deleted for user '
                          f' {iam_username}.')
                    return {'status': 200}
                else:
                    print(
                        'secret manager previous value doesn\'t match with '
                        'inactive IAM key value')
                    return {'status': 400}
            else:
                print('previous key is still active')
        return {'status': 200}
    except ClientError as e:
        print(e)
        return {'status': 500}


def lambda_handler(event, context):
    action = event["action"]
    iam_username = event["username"]
    status = {'status': 500}

    print(f'Detected Action: {action}')
    print(f'Detected IAM username: {iam_username}')

    if action == "create":
        status = create_key(iam_username)
    elif action == "deactivate":
        status = deactive_key(iam_username)
    elif action == "delete":
        status = delete_key(iam_username)

    return status

5. Create Event bridge rule to trigger creating access key.

  • Select “default” event bus
  • Define the pattern to use “Schedule”
    • Set Fixed Rate to every 90 days
  • Set the target as “Lambda function”
    • Set the Function to the lambda function name in step 4
    • Set “Configure input” setting to “Constant (JSON text)”
      • Set value to { “action”: “create”, “username”: “<the iam username in step 1>”
  • Add tags as necessary

6. Create Event bridge rule to trigger deactivating access key.

  • Select “default” event busDefine the pattern to use “Schedule”
  • Set Fixed Rate to every 104 daysSet the target as “Lambda function”
  • Set the Function to the lambda function name in step 4
    • Set “Configure input” setting to “Constant (JSON text)”
      • Set value to { “action”: “deactivate”, “username”: “<the iam username in step 1>”

7. Create Event bridge rule to trigger deleting deactivated access key.

  • Select “default” event busDefine the pattern to use “Schedule”
  • Set Fixed Rate to every 118 daysSet the target as “Lambda function”
  • Set the Function to the lambda function name in step 4
  • Set the Function to the lambda function name in step 4
    • Set “Configure input” setting to “Constant (JSON text)”
      • Set value to { “action”: “delete”, “username”: “<the iam username in step 1>”

8. Create IAM Policy to enabled our lambda function to 1) Access Secrets Manager Secret 3) Access IAM service to manage user access key.

{
    "Version":"2012-10-17",
    "Statement": [
      {
        "Effect": "Allow"
        "Action": [
          "secretsmanager:GetSecretValue",
          "secretsmanager:PutSecretValue"
        ],
        "Resource": "<secrets manager secret arn>"
      },
      {
        "Effect": "Allow"
        "Action": [
          "iam:UpdateAccessKey",
          "iam:CreateAccessKey",
          "iam:DeleteAccessKey",
        ],
        "Resource": "<secrets manager secret arn>"
        "Principal": {
          "AWS": "<iam role arn>"
        }         
      },
      {
        "Effect": "Allow"
        "Action": "iam:ListAccessKeys",
        "Resource": "*"
      },
      {
        "Effect": "Allow"
        "Action": "sns:Publish",
        "Resource": "<sns topic arn>"
      }
    ]
}

9. Create IAM Policy to grant IAM User permissions to access secrets manager secret that stores AWS Access Key and Secret Access Key.

{
    "Version":"2012-10-17",
    "Statement": [
      {
        "Effect": "Allow"
        "Action": [
           <list of actions the API access should grant>
        ],
        "Resource": [
           "<the resources access should be granted to>"
        ],
      },
      {
        "Effect": "Allow"
        "Action": [
          "secretsmanager:GetSecretValue"
        ],
        "Resource": "<secrets manager secret arn>"
        "Principal": {
          "AWS": "<iam role arn>"
        }         
      }
   ]
}

10. Revisit IAM user and attach the new policy created in step 9.

11. Revisit IAM Lambda role and attach the new policy created in step 8.

12. Attach AWS Managed IAM Policy “AWSLambdaBasicExecutionRole” to IAM lambda role.

13. Revisit Sthe secrets manager secret in step 3 and add the following policy to “Resource Permissions”

{
  "Version" : "2012-10-17",
  "Statement" : [ {
    "Sid" : "AllowLambdaFunctionReadWriteAccess",
    "Effect" : "Allow",
    "Principal" : {
      "AWS" : "<lambda iam role>"
    },
    "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue" ],
    "Resource" : "<the arn of the secret>"
  }, {
    "Sid" : "AllowIAMUserReadAccess",
    "Effect" : "Allow",
    "Principal" : {
      "AWS" : "<the arn of the iam user>"
    },
    "Action" : "secretsmanager:GetSecretValue",
    "Resource" : "<the arn of the secret>"
  } ]
}

Conclusion

By using event bridge rules, we can set schedules to trigger the lambda function and pass the event data needed to process key rotation. In our design, we provide the developers with 14 days to rotate their keys and provide an additional 14 days of grace period before deleting the keys permanently. The notification is provided to the software developers via SNS topic and subscribing them to those topics. We use secrets manager to store the secret key and secret access key information; Since secrets manager maintain versions of secret, the lambda function can leverage this to match the last issued access key Id in order to deactivate it during key rotation. IAM user and policy is setup to grant the user access to the access key secret as well as permissions that the software developers needed. IAM role and policy is created to allow the lambda to execute via AWSLambdaBasicExecutionRole managed policy and attaching inline or custom policy to grant access to the access key secret as well as SNS topic to publish notifications to software developers during key rotation and key deactivation.

Coming Soon

In this post, you’ve seen how I build these resources manually using AWS Console. Benhur P.’s post provides you with an overview of how to create the same resources using CloudFormation.

In a follow up post at later date, I will demonstrate how this can be done using Terraform, a tool that I use on a daily basis.

Credits

This post was made possible due to these amazing authors. My ideas were expressed and built on top of their work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s