The Complete AWS Key Management Service (KMS) Guide — CMKs, Envelope Encryption & Best Practice 2025

Imagine storing sensitive customer data—credit card numbers, health records, API secrets—completely unencrypted in the cloud. One misconfigured S3 bucket, one compromised credential, and exposure can be immediate and costly.

That’s exactly why AWS KMS exists.

If you’re building production workloads on AWS, encryption isn’t optional anymore. It’s table stakes. And AWS Key Management Service sits at the center of how encryption actually works across your entire cloud infrastructure.

In this aws kms tutorial, I’ll walk you through everything—from basic concepts to envelope encryption, key policies, and the real-world patterns I’ve seen work (and fail) in production environments. Whether you’re preparing for the AWS SAA-C03, Security Specialty, or DevOps Professional exam, or just trying to lock down your infrastructure properly, this guide has you covered.

Let’s dive in.

What Is AWS KMS and Why Should You Care?

AWS Key Management Service (KMS) is a managed service that lets you create, control, and manage cryptographic keys used to encrypt your data across AWS services.

Think of KMS keys like digital master locks protecting your cloud data. You control who can use those locks, when they can be used, and for what purpose.

Here’s the thing most people miss: KMS isn’t just about “turning on encryption.” It’s about centralized key management with fine-grained access control, automatic rotation, and full audit trails.

Every time you encrypt an S3 object, an RDS database, a Lambda environment variable, or an EBS volume—KMS is likely involved behind the scenes.

Common KMS use cases in DevOps:

• Encrypting S3 objects at rest with server-side encryption
• Protecting RDS database storage and automated backups
• Securing Lambda environment variables containing API keys
• Encrypting EBS volumes attached to EC2 instances
• Managing secrets in AWS Secrets Manager and SSM Parameter Store
• Signing container images and code artifacts

The beauty of KMS? You don’t handle raw cryptographic keys directly. AWS manages the hard parts—key storage, hardware security modules (HSMs), availability—while you focus on defining who gets access to what.

📚 Related Reading: If you’re new to IAM, check out our IAM Policies Deep Dive before continuing. Understanding IAM is essential for mastering KMS access control.

How AWS KMS Works: Architecture Deep Dive

Understanding the KMS architecture helps you make better decisions about key design and access patterns.

Control Plane vs Data Plane

KMS operates with two logical planes:

The Control Plane handles administrative operations—creating keys, setting policies, enabling rotation, managing aliases. These are the operations that define your key infrastructure.

The Data Plane handles cryptographic operations—encrypting, decrypting, generating data keys. These are the high-throughput operations your applications call constantly.

This separation matters for performance. Data plane operations are optimized for low latency and high availability. Control plane operations are less frequent but critical for security governance.

Types of KMS Keys

AWS KMS supports several key types, and choosing the right one depends on your use case:

AWS-managed keys are created and managed by AWS on your behalf. They’re free to store (you pay only for usage) and automatically rotate every year. You’ll see these with aliases like aws/s3, aws/rds, or aws/ebs. Simple, but limited control.

Customer-managed keys (CMKs) give you full control. You define key policies, manage rotation schedules, and can disable or delete keys. These cost $1/month per key plus API usage fees. For production workloads with compliance requirements, CMKs are usually the right choice.

Asymmetric keys support public-key cryptography for encryption or digital signing. Useful when you need to verify signatures outside AWS or share a public key with external parties.

HMAC keys generate and verify hash-based message authentication codes. Great for token validation, API request signing, and data integrity verification.

🎓 SAA-C03 Exam Tip: Know the difference between AWS-managed keys and customer-managed keys. Exam questions often test whether you understand when CMKs are required (cross-account access, custom rotation, key policy control).

Key Components You Need to Know

Every KMS key has several important attributes:

• Key ID: A unique identifier (UUID format) for the key
• Key ARN: The full Amazon Resource Name, used in IAM policies
• Key Material: The actual cryptographic material (either AWS-generated or imported)
• Key Policy: The resource-based policy attached directly to the key
• Aliases: Friendly names like alias/my-app-key for easier reference
• Grants: Temporary permissions for delegated access

Authentication vs Authorization

When your application calls kms:Decrypt, two things happen:

First, AWS verifies authentication—is the caller who they claim to be? This uses IAM credentials, STS tokens, or service roles.

Second, KMS checks authorization—does this authenticated identity have permission to perform this operation on this specific key? This involves evaluating both IAM policies and the key policy.

Both must pass. A mismatch anywhere means access denied.

Customer Managed Keys (CMKs) Explained

For anything beyond basic encryption needs, customer-managed keys are where you want to be.

Why? Because CMKs give you:

• Full control over key policies (who can use and administer the key)
• Custom rotation schedules (or disable rotation entirely for specific use cases)
• The ability to disable or schedule deletion
• Cross-account sharing capabilities
• Integration with CloudTrail for complete audit logging

AWS-managed keys vs Customer-managed keys

AWS-managed keys work fine for simple use cases. But they come with limitations—you can’t change the key policy, can’t share them across accounts, and can’t control rotation beyond the default annual schedule.

With CMKs, you’re in the driver’s seat. You decide exactly which IAM roles can encrypt, which can decrypt, and which can administer the key itself.

Key Lifecycle and States

CMKs move through several states:

• Enabled: Key is active and available for cryptographic operations
• Disabled: Key exists but cannot be used until re-enabled
• Pending Deletion: Key is scheduled for deletion (7-30 day waiting period)
• Pending Import: Key is created but awaiting external key material

Understanding these states matters for operational procedures. You can’t decrypt data if the key that encrypted it is disabled or deleted. Plan accordingly.

Automatic Key Rotation

For symmetric CMKs, you can enable automatic annual rotation. When rotation occurs, AWS generates new key material but keeps the same key ID and ARN. Old key material is retained indefinitely so you can still decrypt data encrypted before rotation.

Important clarification: AWS-managed keys rotate automatically every year. For customer-managed symmetric keys, you must explicitly enable rotation—it’s not on by default. Asymmetric keys and HMAC keys don’t support automatic rotation.

For more details, see the AWS KMS Key Rotation documentation.

Real Example: Creating a KMS Key for S3 and CloudTrail

Here’s how I typically set up a CMK for encrypting both S3 buckets and CloudTrail logs:

# Create the key with descriptive tags
aws kms create-key \
  --description "Production encryption key for S3 and CloudTrail" \
  --tags TagKey=Environment,TagValue=Production TagKey=Owner,TagValue=SecurityTeam \
  --query 'KeyMetadata.KeyId' --output text

# Create a human-friendly alias
aws kms create-alias \
  --alias-name alias/prod-encryption-key \
  --target-key-id <key-id-from-above>

# Enable automatic rotation
aws kms enable-key-rotation --key-id alias/prod-encryption-key

The alias makes it much easier to reference in Terraform, CloudFormation, or application code.

🤔 Reflection Prompt:
Do you need full control over key policies and rotation, or will AWS-managed keys be enough for your workload? Consider compliance requirements, multi-account architectures, and operational overhead.

KMS Policies, Key Policies & Grants

This is where most people get confused. KMS has two layers of authorization: key policies and IAM policies. Both matter.

Key Policies Are Mandatory

Every KMS key has exactly one key policy attached directly to it. This policy is the primary access control mechanism.

Unlike S3 bucket policies (which are optional), key policies are required. If you don’t specify one, AWS creates a default policy that gives the root account full access.

Key Policy Structure

A key policy looks like a standard IAM policy but is attached to the key itself:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Allow administration of the key",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::111122223333:role/KeyAdminRole"},
      "Action": [
        "kms:Create*",
        "kms:Describe*",
        "kms:Enable*",
        "kms:List*",
        "kms:Put*",
        "kms:Update*",
        "kms:Revoke*",
        "kms:Disable*",
        "kms:Delete*",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow use of the key for encryption",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::111122223333:role/AppEncryptionRole"},
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey*"
      ],
      "Resource": "*"
    }
  ]
}

Notice the separation: one statement for key administrators, another for key users. This is the principle of least privilege in action.

Key Policy vs IAM Policy

Here’s the critical rule: by default, IAM policies alone are not enough to grant KMS access.

The key policy must either grant access directly OR include a statement allowing IAM policies to grant access. That’s why the default key policy includes this statement:

{
  "Sid": "Enable IAM policies",
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::111122223333:root"},
  "Action": "kms:*",
  "Resource": "*"
}

Without this, IAM policies can’t grant any KMS permissions—even if they say "Allow".

Policy Evaluation Logic

The evaluation order matters:

1. If the key policy explicitly denies → Access denied
2. If the key policy allows AND IAM policy allows → Access granted
3. If only one allows (without the “Enable IAM policies” statement) → Access denied

Explicit deny always wins, regardless of where it appears.

Grants for Temporary Access

Grants provide a way to delegate permissions without modifying the key policy. They’re perfect for:

• Allowing AWS services to use keys on your behalf
• Temporary access that automatically expires
• Cross-service encryption workflows

When you encrypt an EBS volume, AWS creates a grant behind the scenes allowing the EC2 service to decrypt the volume on boot.

📝 Quiz Check:
What happens if an IAM policy allows kms:Decrypt but the key policy explicitly denies access for that principal?
Click to reveal answer

Answer: Access is denied. Explicit deny in any policy always takes precedence over any allow statement. This is fundamental to AWS policy evaluation—deny wins.

Envelope Encryption: The Heart of KMS

This is the concept that trips up most people studying for AWS certifications—but once it clicks, everything makes sense.

The Problem with Direct Encryption

You could theoretically send all your data to KMS for encryption. But that creates two problems:

1. Performance: Network round-trips for every encryption operation add latency
2. Data size limits: KMS can only encrypt up to 4,096 bytes (4 KB) directly via the Encrypt API (see AWS KMS Encrypt API documentation)

The Envelope Encryption Solution

Envelope encryption solves both problems elegantly:

Step 1: Your application calls kms:GenerateDataKey. KMS returns two things—a plaintext data key and an encrypted copy of that same key (encrypted with your CMK).

Step 2: Your application uses the plaintext data key to encrypt your actual data locally. This happens on your server, not in KMS.

Step 3: Your application stores the encrypted data alongside the encrypted data key (called the ciphertext blob).

Step 4: Immediately discard the plaintext data key from memory. Never store it.

Decryption Flow

When you need to read the data:

Step 1: Retrieve the encrypted data and the encrypted data key from storage.

Step 2: Call kms:Decrypt with the encrypted data key. KMS returns the plaintext data key.

Step 3: Use the plaintext data key to decrypt your data locally.

Step 4: Discard the plaintext data key from memory.

GenerateDataKeyWithoutPlaintext: When to Use It

There’s an alternative API—GenerateDataKeyWithoutPlaintext—that returns only the encrypted data key, never the plaintext. Why would you want this?

Use it when your application generates the key but a different service or system will perform the actual encryption/decryption. This minimizes plaintext key exposure in your application tier. For example, if you’re setting up encryption for a downstream service that will handle decryption itself, you never need to see the plaintext key.

For details, see the GenerateDataKeyWithoutPlaintext API documentation.

Hands-On Lab: Envelope Encryption with CLI and Python

Let’s walk through a complete envelope encryption workflow. This is the kind of hands-on practice that cements understanding.

Step 1: Generate a Data Key (CLI)

# Generate a data key and capture both plaintext and encrypted versions
aws kms generate-data-key \
  --key-id alias/prod-encryption-key \
  --key-spec AES_256 \
  --output json > datakey_response.json

# Extract the plaintext key (base64 encoded)
cat datakey_response.json | jq -r '.Plaintext' | base64 --decode > plaintext_datakey.bin

# Extract the encrypted key (for storage)
cat datakey_response.json | jq -r '.CiphertextBlob' | base64 --decode > encrypted_datakey.bin

# Verify key sizes
ls -la *.bin
# plaintext_datakey.bin should be 32 bytes (256 bits)
# encrypted_datakey.bin will be larger due to KMS metadata

Step 2: Encrypt Data Locally (Python)

import boto3
import base64
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import os

def envelope_encrypt(plaintext_data: bytes, key_alias: str) -> dict:
    """
    Encrypt data using envelope encryption pattern.
    Returns encrypted data and encrypted data key for storage.
    """
    kms = boto3.client('kms')

    # Step 1: Get data key from KMS
    response = kms.generate_data_key(
        KeyId=key_alias,
        KeySpec='AES_256'
    )

    plaintext_key = response['Plaintext']      # 32 bytes for AES-256
    encrypted_key = response['CiphertextBlob'] # Store this with your data

    # Step 2: Encrypt data locally using AES-GCM
    # Generate a random 96-bit nonce (required for AES-GCM)
    nonce = os.urandom(12)

    # Create cipher and encrypt
    aesgcm = AESGCM(plaintext_key)
    ciphertext = aesgcm.encrypt(nonce, plaintext_data, None)

    # Step 3: CRITICAL - Clear plaintext key from memory
    # In production, use secure memory wiping
    del plaintext_key

    # Return everything needed for decryption
    return {
        'encrypted_data': base64.b64encode(nonce + ciphertext).decode(),
        'encrypted_key': base64.b64encode(encrypted_key).decode()
    }

def envelope_decrypt(encrypted_payload: dict) -> bytes:
    """
    Decrypt data using envelope encryption pattern.
    """
    kms = boto3.client('kms')

    # Decode the stored values
    encrypted_key = base64.b64decode(encrypted_payload['encrypted_key'])
    encrypted_blob = base64.b64decode(encrypted_payload['encrypted_data'])

    # Extract nonce (first 12 bytes) and ciphertext
    nonce = encrypted_blob[:12]
    ciphertext = encrypted_blob[12:]

    # Step 1: Decrypt the data key using KMS
    response = kms.decrypt(CiphertextBlob=encrypted_key)
    plaintext_key = response['Plaintext']

    # Step 2: Decrypt data locally
    aesgcm = AESGCM(plaintext_key)
    plaintext_data = aesgcm.decrypt(nonce, ciphertext, None)

    # Step 3: Clear plaintext key
    del plaintext_key

    return plaintext_data

# Usage example
if __name__ == "__main__":
    # Encrypt sensitive user data
    user_pii = b'{"ssn": "123-45-6789", "dob": "1990-01-15"}'

    encrypted = envelope_encrypt(user_pii, 'alias/user-pii-key')
    print(f"Encrypted key length: {len(encrypted['encrypted_key'])}")
    print(f"Encrypted data length: {len(encrypted['encrypted_data'])}")

    # Later, decrypt it
    decrypted = envelope_decrypt(encrypted)
    print(f"Decrypted: {decrypted.decode()}")

Step 3: Decrypt the Data Key (CLI)

# When you need to decrypt, first recover the plaintext data key
aws kms decrypt \
  --ciphertext-blob fileb://encrypted_datakey.bin \
  --query 'Plaintext' \
  --output text | base64 --decode > recovered_plaintext_key.bin

# Verify it matches the original
diff plaintext_datakey.bin recovered_plaintext_key.bin
# Should show no difference

# IMPORTANT: Delete plaintext keys after use
rm -f plaintext_datakey.bin recovered_plaintext_key.bin

💡 How I Use This in Production:
In one project, we encrypted patient health records before storing them in S3. The envelope encryption pattern let us process thousands of records per second locally while only making KMS calls for key generation and decryption. Our KMS costs stayed under $50/month even with millions of encrypted objects.

Cross-Account KMS Access Patterns

In multi-account AWS environments, sharing KMS keys across accounts is common—and requires careful configuration on both sides.

The Two-Sided Handshake

Cross-account KMS access requires permissions in two places:

1. The key policy in the key-owning account must allow the external account
2. The IAM policy in the consuming account must allow the principal to use the key

Step 1: Key Policy in Account A (Key Owner)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountUse",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::999988887777:root"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowCrossAccountGrant",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::999988887777:root"
      },
      "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": "*",
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    }
  ]
}

Step 2: IAM Role and Policy in Account B (Consumer)

First, create an IAM role with a trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Then attach an IAM policy allowing KMS operations on the cross-account key:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountKMSUse",
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey*"
      ],
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/12345678-1234-1234-1234-123456789012"
    }
  ]
}

Step 3: CLI Sequence to Test Cross-Account Access

From Account B, assuming the role and testing encryption:

# Assume the role in Account B
export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
  $(aws sts assume-role \
    --role-arn arn:aws:iam::999988887777:role/CrossAccountKMSRole \
    --role-session-name test-session \
    --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
    --output text))

# Test encryption using Account A's key
echo "test data" | aws kms encrypt \
  --key-id arn:aws:kms:us-east-1:111122223333:key/12345678-1234-1234-1234-123456789012 \
  --plaintext fileb:///dev/stdin \
  --query CiphertextBlob \
  --output text

# If successful, you'll see base64-encoded ciphertext

KMS Best Practices for Production

After years of working with KMS in production, here’s what actually matters:

Enable automatic key rotation for all symmetric CMKs. AWS rotates the key material annually while keeping the same key ID, so your applications don’t need changes. One-line configuration that significantly improves security posture.

Use CMKs for sensitive data, not AWS-managed keys. The extra dollar per month gives you control over policies, cross-account access, and compliance audit trails.

Never store plaintext data keys—anywhere. Not in logs, not in databases, not in environment variables. Generate, use, discard.

Integrate KMS with CloudTrail and actually review the logs. Every Encrypt, Decrypt, and GenerateDataKey call is recorded. Set up alerts for unusual patterns.

Use Grants instead of overly permissive policies. Need temporary access? Grants can be revoked instantly without modifying key policies.

Separate key administrators from key users. The person who can delete a key shouldn’t necessarily be able to decrypt data with it.

Tag keys using a defined strategy. Tags like Environment, CostCenter, DataClassification, and Owner make governance and cost allocation much easier.

Use multi-region keys for disaster recovery. If your DR strategy involves another region, multi-region keys ensure encrypted data remains accessible after failover.

🚀 Security Tip:
Avoid embedding encrypted values directly in EC2 user data scripts. Instead, store sensitive values in SSM Parameter Store as SecureString and retrieve them at runtime. This prevents secrets from appearing in instance metadata or CloudFormation templates.
Related: SSM Parameter Store Best Practices

KMS Monitoring, Logging & Incident Response

You can’t secure what you can’t see. Here’s how to maintain visibility into KMS operations—and what to do when things go wrong.

CloudTrail Integration

Every KMS API call generates a CloudTrail event. Key events to monitor:

• Decrypt calls from unexpected principals
• DisableKey or ScheduleKeyDeletion events
• CreateGrant operations (could indicate privilege escalation)
• PutKeyPolicy changes (policy modifications)

💡 Tip — CloudTrail + CloudWatch Alert:
Create CloudWatch metric filters for KMS CloudTrail events like Decrypt, DisableKey, and ScheduleKeyDeletion, and alert on spikes or cross-account decrypt events. See KMS CloudTrail logging documentation.

AWS Config Rules

Enable Config rules to enforce KMS compliance:

• kms-cmk-not-scheduled-for-deletion — Alert on keys pending deletion
• cmk-backing-key-rotation-enabled — Ensure rotation is enabled
• Custom rules for key policy requirements

CloudWatch Alarms

Set up alarms for:

• High error rates on decrypt operations (could indicate attacks or misconfigurations)
• Unusual API call volumes
• Cross-account access patterns

GuardDuty Findings

GuardDuty can detect anomalous KMS usage patterns, including:

• Access from unusual IP addresses or locations
• API calls from known malicious infrastructure
• Potential credential compromise scenarios

Incident Response Runbook: KMS Issues

When KMS problems occur in production, having a runbook ready saves critical time.

Scenario 1: Key Accidentally Disabled

Symptoms: Applications throwing DisabledException errors, decryption failures.

Response:

# 1. Identify the disabled key
aws kms list-keys --query 'Keys[*].KeyId' --output text | \
  xargs -I {} aws kms describe-key --key-id {} \
  --query 'KeyMetadata.[KeyId,KeyState]' --output text | grep -i disabled

# 2. Re-enable the key (requires kms:EnableKey permission)
aws kms enable-key --key-id <key-id>

# 3. Verify
aws kms describe-key --key-id <key-id> --query 'KeyMetadata.KeyState'
# Should return "Enabled"

Scenario 2: Unexpected Decrypt Spikes

Symptoms: CloudWatch showing 10x normal decrypt volume, potential cost impact or security incident.

Investigation:

# 1. Query CloudTrail for decrypt events in the last hour
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ) \
  --query 'Events[*].[EventTime,Username,Resources[0].ResourceName]' \
  --output table

# 2. Identify unusual source IPs or principals
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ) \
  --query 'Events[*].CloudTrailEvent' --output text | \
  jq -r '.sourceIPAddress' | sort | uniq -c | sort -rn | head -10

Response actions:

1. If malicious: Disable the compromised credentials immediately via IAM
2. If misconfigured application: Fix the code (often a loop calling decrypt unnecessarily)
3. Document the incident and update monitoring thresholds

Scenario 3: Key Pending Deletion

Symptoms: Key showing PendingDeletion state, 7-30 day countdown active.

Response:

# 1. Cancel deletion if the key is still needed
aws kms cancel-key-deletion --key-id <key-id>

# 2. Re-enable the key (canceling deletion leaves it disabled)
aws kms enable-key --key-id <key-id>

# 3. Investigate who scheduled deletion
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=ScheduleKeyDeletion \
  --query 'Events[*].[EventTime,Username]' --output table

🤔 Reflection:
How often are your KMS keys reviewed and rotated in your current environment? Is there a defined process, or does it happen ad-hoc? Consider adding key reviews to your quarterly security checklist.

Integrating KMS with AWS Services

KMS integrates seamlessly with dozens of AWS services. Here’s how the most common ones work:

S3 Server-Side Encryption (SSE-KMS)

When you enable SSE-KMS on a bucket, S3 calls GenerateDataKey for each object and stores the encrypted data key alongside the object. Decryption happens transparently when authorized users download.

EBS Volume Encryption

Encrypted EBS volumes use envelope encryption. AWS creates a grant allowing EC2 to decrypt the volume key on your behalf during instance launch.

RDS Encryption at Rest

RDS encrypts database storage, automated backups, snapshots, and read replicas. Once enabled, you can’t disable it—plan encryption from day one.

Lambda Environment Variables

Lambda encrypts environment variables using KMS. You can use the default service key or specify a CMK for additional control.

EKS Secrets Encryption

Kubernetes secrets in EKS can be encrypted using a CMK, adding a layer of protection beyond etcd encryption.

SSM Parameter Store SecureString

Parameters stored as SecureString are encrypted with KMS. Perfect for database passwords, API keys, and configuration secrets.

Secrets Manager

Every secret in Secrets Manager is encrypted with KMS. Automatic rotation? Secrets Manager handles the decryption and re-encryption automatically.

📚 Related: Terraform KMS Integration Guide | Secrets Manager Deep Dive

Example: Encrypting Terraform Remote State

resource "aws_kms_key" "terraform_state" {
  description             = "Terraform state encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  tags = {
    Environment = "Production"
    ManagedBy   = "Terraform"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.terraform_state.arn
      sse_algorithm     = "aws:kms"
    }
    bucket_key_enabled = true  # Important for cost optimization!
  }
}

🤔 Reflection:
Would you choose a CMK or an AWS-managed key when building a multi-account CI/CD pipeline? Consider cross-account access requirements and operational complexity.

Advanced KMS Features

For complex architectures, these advanced features become essential.

Multi-Region Keys

Multi-region keys are replicas that share the same key material across AWS regions. A key encrypted in us-east-1 can be decrypted in eu-west-1 using the replica—without cross-region API calls.

Critical for disaster recovery and globally distributed applications.

Real-world scenario: Your primary application runs in us-east-1, encrypting database backups. When you fail over to eu-west-1, the replica key lets you decrypt those backups immediately without any key management changes.

Important: Each primary and replica multi-region key is billed separately. A key replicated to 3 regions costs $3/month plus individual API charges per region. See Multi-Region Keys documentation.

Asymmetric Keys

Asymmetric keys support RSA and ECC algorithms for:

• Digital signature verification (code signing, document signing)
• Encryption outside AWS (share public key externally)
• JWT token signing

Real-world scenario: Your CI/CD pipeline signs container images using a KMS asymmetric key. External systems can verify signatures using the public key without AWS access.

# Create an asymmetric signing key
aws kms create-key \
  --key-spec RSA_2048 \
  --key-usage SIGN_VERIFY \
  --description "Container image signing key"

# Get the public key for external distribution
aws kms get-public-key --key-id <key-id> \
  --query 'PublicKey' --output text | base64 --decode > public_key.der

HMAC Keys

HMAC keys generate hash-based message authentication codes for:

• API request authentication
• Token validation
• Data integrity verification

Real-world scenario: Your API gateway validates incoming webhook signatures using a KMS HMAC key, ensuring requests actually come from your payment provider.

# Generate HMAC for a webhook payload
aws kms generate-mac \
  --key-id alias/webhook-hmac-key \
  --mac-algorithm HMAC_SHA_256 \
  --message fileb://webhook_payload.json \
  --query 'Mac' --output text

External Key Stores (XKS)

For organizations requiring keys stored in their own HSMs outside AWS, External Key Store (XKS) allows KMS to communicate with external key managers. This satisfies strict compliance requirements (FIPS 140-2 Level 3, specific regulatory mandates) while maintaining KMS integration benefits.

XKS workflow:

1. You deploy an XKS proxy in your environment
2. The proxy connects to your external HSM
3. KMS routes cryptographic operations through the proxy to your HSM
4. Key material never enters AWS

This is complex to set up and operate—use it only when compliance absolutely requires it.

Key Material Import

You can import your own key material instead of letting AWS generate it. Useful for specific compliance scenarios, but adds operational complexity—you’re responsible for key material backup and durability.

Common KMS Mistakes to Avoid

I’ve seen these mistakes cause real production issues:

Using AWS-managed keys for sensitive production data limits your ability to control access, rotate on custom schedules, or share across accounts. Spend the dollar per month for CMKs.

Granting kms:* permissions is lazy and dangerous. Be specific about which actions each principal needs.

Not enabling key rotation leaves you with the same key material forever. Enable rotation on every symmetric CMK unless you have a documented reason not to.

Leaving keys in “pending deletion” limbo without review. Set up alerts for this state and investigate before the deletion window expires.

Storing encrypted data keys in application logs defeats the purpose of encryption. Audit your logging to ensure plaintext keys never appear.

Using one CMK for everything creates a blast radius problem. If that key is compromised or accidentally deleted, all your encrypted data is affected.

Allowing root account to perform encryption operations in production is unnecessary. Delegate to specific roles with minimal required permissions.

⚠️ Real Incident (Anonymized):
A client’s production database became inaccessible after an engineer accidentally disabled a KMS key during “cleanup.” Because they had no CloudTrail alerts configured, the issue wasn’t discovered for 6 hours—when the overnight batch job failed. The fix took 30 seconds (re-enable the key), but the incident cost them an SLA breach. Lesson: Always set up DisableKey and ScheduleKeyDeletion alerts.

KMS Pricing: What You’ll Actually Pay

KMS isn’t free, and costs can add up in high-volume environments.

Pricing as of 2025 (always verify at AWS KMS Pricing):

Key Storage

Customer-managed keys cost $1.00 per month per key (prorated hourly). AWS-managed keys have no storage charge.

API Request Costs

Symmetric key operations cost $0.03 per 10,000 requests. This includes Encrypt, Decrypt, GenerateDataKey, GenerateDataKeyWithoutPlaintext, and ReEncrypt.

Asymmetric operations (RSA, ECC) cost $0.10 – $0.15 per 10,000 requests depending on key type and operation.

Some calls are free: DescribeKey, GetKeyPolicy, ListKeys, ListAliases.

Multi-Region Key Costs

Each replica is billed as a separate key ($1/month each), and API calls are billed in each region where they occur.

Real-World Cost Example

A Lambda function that decrypts configuration on every cold start:

• 1 million invocations/month
• 100,000 cold starts (assume 10% cold start rate)
• 100,000 decrypt calls × $0.03/10,000 = $0.30/month

A high-throughput application decrypting every request:

• 100 million requests/month
• 100 million decrypt calls × $0.03/10,000 = $300/month

S3 Bucket Keys: Dramatic Cost Savings

Without bucket keys, S3 makes a KMS API call for every object operation. With bucket keys enabled, S3 uses a bucket-level data key and significantly reduces KMS calls.

Real comparison:

• Without bucket keys: 10 million S3 PUTs/month = 10 million GenerateDataKey calls = $30/month
• With bucket keys: Same 10 million PUTs ≈ 1,000 GenerateDataKey calls = $0.003/month

That’s a 99.99% cost reduction for high-volume S3 workloads. Always enable bucket keys for SSE-KMS buckets.

# Enable bucket keys on existing bucket
aws s3api put-bucket-encryption \
  --bucket my-bucket \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "alias/my-key"
      },
      "BucketKeyEnabled": true
    }]
  }'

FAQ: AWS KMS Common Questions

Conclusion

KMS isn’t just another AWS service to check off your security list. It’s the backbone of encryption across your entire cloud infrastructure.

When you understand envelope encryption, master key policies, and implement proper access controls, you’ve solved one of the hardest problems in cloud security—managing cryptographic keys at scale without operational nightmare.

The patterns we covered—CMKs for control, envelope encryption for performance, CloudTrail for visibility—these are the foundations of production-ready encryption.

Start with your most sensitive workloads. Enable CMKs, configure proper policies, turn on rotation, and set up monitoring. Then expand from there.

Secure your keys, and you secure your entire cloud.

👉 Ready to go deeper? Take the free AWS KMS Crash Course—Learn encryption, key policies, rotation, and envelope encryption through hands-on labs and real-world scenarios.

About the Author

Srikanth Ch is a Senior DevOps Engineer with extensive experience in AWS architecture, Kubernetes, and CI/CD pipelines. He holds AWS Solutions Architect and Security Specialty certifications and has helped organizations implement enterprise-grade encryption strategies across multi-account AWS environments. Srikanth writes at thedevopstooling.com to share practical DevOps knowledge with the community.

References

• AWS KMS Developer Guide
• AWS KMS Pricing
• KMS Encrypt API Documentation
• GenerateDataKeyWithoutPlaintext API
• Multi-Region Keys Overview
• KMS Key Rotation
• KMS CloudTrail Logging

[thedevopstooling.com — Your path to DevOps mastery]