Troubleshooting - IAM Audit

Overview

This guide covers common errors you may encounter when running iam-audit, along with diagnostic steps and solutions. Most errors are related to AWS permissions, role configuration, or API throttling.

Permission Errors

AccessDenied: Not authorized to perform sts:AssumeRole

Error Message

Error en cuenta ProductionAccount: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam::111111111111:user/auditor is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/IAMAuditRole

Cause

The credentials from your --profile cannot assume the specified --role in the target account.

Diagnosis

1. Verify the role exists in the target account:

aws iam get-role --role-name IAMAuditRole --profile <target-account-profile>

Expected output:

{
    "Role": {
        "RoleName": "IAMAuditRole",
        "Arn": "arn:aws:iam::222222222222:role/IAMAuditRole"
    }
}

2. Check the trust policy of the target role:

aws iam get-role --role-name IAMAuditRole --profile <target-account-profile> --query 'Role.AssumeRolePolicyDocument'

The trust policy must allow the management account to assume it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

3. Verify your management account profile has AssumeRole permission:

aws iam get-user --profile <management-profile>
aws iam list-attached-user-policies --user-name <your-user> --profile <management-profile>

Solution

Option 1: Fix the trust policy in the target account

aws iam update-assume-role-policy \
  --role-name IAMAuditRole \
  --policy-document file://trust-policy.json \
  --profile <target-account-profile>

trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Option 2: Deploy the role using CloudFormation StackSets from the management account

aws cloudformation create-stack-set \
  --stack-set-name IAMAuditRole \
  --template-body file://audit-role-template.yaml \
  --permission-model SERVICE_MANAGED \
  --auto-deployment Enabled=true

AccessDenied: organizations:ListAccounts

Error Message

An error occurred (AccessDeniedException) when calling the ListAccounts operation: You don't have permissions to access this resource.

Cause

The AWS profile specified with --profile does not have permission to call organizations:ListAccounts.

Diagnosis

aws organizations list-accounts --profile <management-profile>

Solution

Attach this policy to your IAM user/role in the management account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "organizations:ListAccounts",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::*:role/IAMAuditRole"
    }
  ]
}

Apply it:

aws iam put-user-policy \
  --user-name <your-user> \
  --policy-name IAMAuditPermissions \
  --policy-document file://audit-permissions.json \
  --profile <management-profile>

NoSuchEntity: The user with name X cannot be found

Error Message

An error occurred (NoSuchEntity) when calling the GetLoginProfile operation: The user with name testuser cannot be found.

Cause

This is not an error. It’s expected behavior when a user does not have console access (no login profile).

How It’s Handled

The script catches this exception:

try:
    password_response = iam_client.get_login_profile(UserName=user['UserName'])
    password_status = 'Configurada'
except iam_client.exceptions.NoSuchEntityException:
    password_status = 'No configurada'

Source: iam_audit.py:54-58

Solution

No action needed. This is normal operation.

Role Assumption Failures

Error assuming role: The security token included in the request is invalid

Error Message

An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.

Cause

The AWS credentials for your --profile are invalid, expired, or missing.

Diagnosis

1. Check if credentials are configured:

aws configure list --profile <management-profile>

Expected output:

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                mgmt-profile           None    None
access_key     ******************** shared-credentials-file    
secret_key     ******************** shared-credentials-file    
    region                us-east-1      config-file    ~/.aws/config

2. Test credentials:

aws sts get-caller-identity --profile <management-profile>

Expected output:

{
    "UserId": "AIDAEXAMPLE",
    "Account": "111111111111",
    "Arn": "arn:aws:iam::111111111111:user/auditor"
}

Solution

Option 1: Reconfigure the profile

aws configure --profile <management-profile>

Option 2: If using SSO, refresh the session

aws sso login --profile <management-profile>

Option 3: If using MFA, generate a session token

aws sts get-session-token \
  --serial-number arn:aws:iam::111111111111:mfa/your-user \
  --token-code 123456 \
  --profile <management-profile>

Then update ~/.aws/credentials with the temporary credentials.

Role assumption succeeds but IAM calls fail

Error Message

Error en cuenta DevAccount: An error occurred (AccessDenied) when calling the ListUsers operation: User: arn:aws:sts::222222222222:assumed-role/IAMAuditRole/SecurityAudit is not authorized to perform: iam:ListUsers on resource: arn:aws:iam::222222222222:user/

Cause

The assumed role exists but lacks the required IAM permissions.

Diagnosis

Check the role’s attached policies:

aws iam list-attached-role-policies --role-name IAMAuditRole --profile <target-account>

Check inline policies:

aws iam list-role-policies --role-name IAMAuditRole --profile <target-account>

Solution

Attach the required permissions to the audit role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:ListUsers",
        "iam:ListAccessKeys",
        "iam:GetAccessKeyLastUsed",
        "iam:ListMFADevices",
        "iam:GetLoginProfile",
        "cloudtrail:LookupEvents"
      ],
      "Resource": "*"
    }
  ]
}

Apply it:

aws iam put-role-policy \
  --role-name IAMAuditRole \
  --policy-name IAMAuditPermissions \
  --policy-document file://iam-audit-permissions.json \
  --profile <target-account>

CloudTrail Errors

Error querying CloudTrail: AccessDenied on cloudtrail:LookupEvents

Error Message

Error en CloudTrail cuenta ProductionAccount: An error occurred (AccessDeniedException) when calling the LookupEvents operation: User: arn:aws:sts::222222222222:assumed-role/IAMAuditRole/CloudTrailAudit is not authorized to perform: cloudtrail:LookupEvents

Cause

The audit role lacks cloudtrail:LookupEvents permission.

Solution

Add the permission to the role policy:

{
  "Effect": "Allow",
  "Action": "cloudtrail:LookupEvents",
  "Resource": "*"
}

Apply:

aws iam put-role-policy \
  --role-name IAMAuditRole \
  --policy-name CloudTrailLookup \
  --policy-document file://cloudtrail-permission.json \
  --profile <target-account>

No CloudTrail events found (Total: 0)

Observation

Total de eventos CloudTrail encontrados: 0

Possible Causes

No IAM events occurred in the date range
CloudTrail is not enabled in the accounts
Events are older than CloudTrail retention (90 days default)
The hardcoded start date is too recent (iam_audit.py:137)

Diagnosis

1. Check if CloudTrail is enabled:

aws cloudtrail describe-trails --profile <account-profile>

2. Manually query CloudTrail:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateUser \
  --max-results 10 \
  --region us-east-1 \
  --profile <account-profile>

3. Check the date range in the script:

start_date = datetime(2026, 2, 18)  # Is this too recent?
end_date = datetime.now()

Solution

Option 1: Modify the start date in iam_audit.py:

start_date = datetime.now() - timedelta(days=90)  # Last 90 days

Option 2: Enable CloudTrail in all accounts:

aws cloudtrail create-trail \
  --name organization-audit-trail \
  --s3-bucket-name <your-cloudtrail-bucket> \
  --is-multi-region-trail \
  --is-organization-trail \
  --profile <management-account>

API Throttling

Throttling: Rate exceeded (IAM API)

Error Message

An error occurred (Throttling) when calling the GetAccessKeyLastUsed operation: Rate exceeded

Cause

Too many IAM API calls in a short time. The script makes 4 API calls per IAM user:

list_access_keys
list_mfa_devices
get_login_profile
get_access_key_last_used

In accounts with hundreds of IAM users, rate limits are exceeded.

Solution

Option 1: Add exponential backoff (recommended)Modify the script to use adaptive retry mode:

from botocore.config import Config

config = Config(
    retries={
        'max_attempts': 10,
        'mode': 'adaptive'  # Automatically backs off
    }
)

iam_client = boto3.client(
    'iam',
    config=config,
    aws_access_key_id=credentials['AccessKeyId'],
    aws_secret_access_key=credentials['SecretAccessKey'],
    aws_session_token=credentials['SessionToken']
)

Option 2: Add manual delays between accounts

import time

for account in accounts:
    print(f"Auditando cuenta: {account['name']}")
    findings = get_iam_users_with_keys(iam_client, account['id'], account['name'])
    all_findings.extend(findings)
    time.sleep(2)  # 2-second delay between accounts

Option 3: Run during off-peak hoursSchedule the script to run when API load is lower (e.g., weekends, nights).

Throttling: Rate exceeded (CloudTrail API)

Error Message

Error en CloudTrail cuenta DevAccount: An error occurred (ThrottlingException) when calling the LookupEvents operation: Rate exceeded

Cause

CloudTrail lookup_events is limited to 2 requests per second per account.The script queries 5 event types per account, which can trigger throttling in large organizations.

Solution

Option 1: Add delays between event queries

import time

for event_name in event_names:
    paginator = ct_client.get_paginator('lookup_events')
    for page in paginator.paginate(...):
        # Process events
    time.sleep(0.6)  # 600ms delay = ~1.6 requests/sec

Option 2: Use boto3 adaptive retry mode (same as IAM throttling solution)Option 3: Reduce the number of monitored events

event_names = ['DeleteAccessKey', 'CreateAccessKey']  # Only critical events

Empty or Missing Report

No CSV file generated

Observation

No se encontraron Access Keys.

No iam_audit_report_*.csv file is created.

Cause

The findings list is empty. This happens when:

No IAM users have access keys (ideal state)
All accounts were skipped due to role assumption errors
Organizations API call failed

Diagnosis

1. Check if accounts were processed:Look for console output like:

Auditando cuenta: ProductionAccount (123456789012)

If you only see error messages, all accounts were skipped.2. Manually verify IAM users exist:

aws iam list-users --profile <account-profile>

3. Check Organizations access:

aws organizations list-accounts --profile <management-profile>

Solution

If accounts were skipped: Fix role assumption errors (see Permission Errors section)If no IAM users have keys: This is good! It means you’re following AWS best practices (use roles, not access keys).To verify the script works: Create a test IAM user with an access key in one account:

aws iam create-user --user-name test-audit-user --profile <account>
aws iam create-access-key --user-name test-audit-user --profile <account>

Run the audit again. You should see the test user in the report.

CSV file is empty (only headers)

Observation

The CSV file exists but contains only headers:

account_id,account_name,username,password_status,password_last_used,access_key_id,status,created_date,last_used_date,service_name,mfa_status

Cause

No IAM users with access keys were found across all accounts.

Solution

This is likely the correct result. Modern AWS best practices recommend:

✅ Use IAM roles with temporary credentials (STS)
✅ Use AWS IAM Identity Center (SSO)
❌ Avoid long-term access keys

If you expect to find keys but the report is empty:

Verify the script ran on all accounts (check console output)
Manually check a few accounts for IAM users:

aws iam list-users --profile <account>

Profile and Configuration Issues

ProfileNotFound: The config profile could not be found

Error Message

botocore.exceptions.ProfileNotFound: The config profile (mgmt-profile) could not be found

Cause

The AWS CLI profile specified with --profile does not exist in your AWS config.

Diagnosis

aws configure list-profiles

Check if your profile is listed.

Solution

Option 1: Create the profile

aws configure --profile mgmt-profile

Option 2: Use an existing profile

python iam_audit.py --profile default --role IAMAuditRole

Option 3: If using SSO, configure the SSO profile

aws configure sso

RegionNotFound: You must specify a region

Error Message

botocore.exceptions.NoRegionError: You must specify a region.

Cause

Your AWS profile does not have a default region configured.

Solution

Option 1: Set region in the profile

aws configure set region us-east-1 --profile <management-profile>

Option 2: Set environment variable

export AWS_DEFAULT_REGION=us-east-1
python iam_audit.py --profile mgmt-profile --role IAMAuditRole

Script Execution Errors

ModuleNotFoundError: No module named 'boto3'

Error Message

ModuleNotFoundError: No module named 'boto3'

Cause

The boto3 library is not installed in your Python environment.

Solution

pip install boto3

Or with a specific Python version:

python3 -m pip install boto3

Verify installation:

python3 -c "import boto3; print(boto3.__version__)"

Script hangs or runs indefinitely

Observation

The script appears to freeze after:

Auditando cuenta: ProductionAccount (123456789012)

Cause

Likely API throttling without proper retry logic, or the script is processing a large number of IAM users.

Diagnosis

Add verbose logging:

import logging
logging.basicConfig(level=logging.INFO)

Or monitor boto3 API calls:

import boto3
boto3.set_stream_logger('boto3.resources', logging.DEBUG)

Solution

Option 1: Add a timeout to boto3 client

from botocore.config import Config

config = Config(
    connect_timeout=10,
    read_timeout=30
)

iam_client = boto3.client('iam', config=config, ...)

Option 2: Run the script with a process timeout

timeout 1800 python iam_audit.py --profile mgmt --role IAMAuditRole

(Kills the process after 30 minutes)

Getting Help

Check AWS CloudTrail Logs

If you’re still encountering issues, check CloudTrail for API call errors:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --max-results 50 \
  --region us-east-1 \
  --profile <management-profile>

Look for events with errorCode or errorMessage fields.

Enable Debug Logging

Modify the script to enable debug output:

import logging
logging.basicConfig(level=logging.DEBUG)

Run the script and capture output:

python iam_audit.py --profile mgmt --role IAMAuditRole 2>&1 | tee debug.log

Community Support

For issues not covered in this guide, open a GitHub issue with:

Full error message
AWS region
Number of accounts in your organization
Relevant CloudTrail logs

GitHub Issues: [Link to repository issues page]

AWS Support Resources

Quick Diagnostic Checklist

Before opening an issue, verify:

AWS CLI profile exists: aws configure list-profiles
Profile has valid credentials: aws sts get-caller-identity --profile <profile>
Management account can list accounts: aws organizations list-accounts --profile <profile>
Target role exists: aws iam get-role --role-name <role> --profile <target-account>
Trust policy allows management account: Check AssumeRolePolicyDocument
Role has required IAM permissions: Check attached policies
boto3 is installed: python3 -c "import boto3"
Python version is 3.9+: python3 --version

All checks passed but still failing? Check the Limitations page for known constraints.

Get Started

Configuration

Usage

Guides

Reference

​Overview

​Permission Errors

​Error Message

​Cause

​Diagnosis

​Solution

​Error Message

​Cause

​Diagnosis

​Solution

​Error Message

​Cause

​How It’s Handled

​Solution

​Role Assumption Failures

​Error Message

​Cause

​Diagnosis

​Solution

​Error Message

​Cause

​Diagnosis

​Solution

​CloudTrail Errors

​Error Message

​Cause

​Solution

​Observation

​Possible Causes

​Diagnosis

​Solution

​API Throttling

​Error Message

​Cause

​Solution

​Error Message

​Cause

​Solution

​Empty or Missing Report

​Observation

​Cause

​Diagnosis

​Solution

​Observation

​Cause

​Solution

​Profile and Configuration Issues

​Error Message

​Cause

​Diagnosis

​Solution

​Error Message

​Cause

​Solution

​Script Execution Errors

​Error Message

​Cause

​Solution

​Observation

​Cause

​Diagnosis

​Solution

​Getting Help

​Check AWS CloudTrail Logs

​Enable Debug Logging

​Community Support

​AWS Support Resources

​Quick Diagnostic Checklist

Build docs developers (and LLMs) love

Overview

Permission Errors

Error Message

Cause

Diagnosis

Solution

Error Message

Cause

Diagnosis

Solution

Error Message

Cause

How It’s Handled

Solution

Role Assumption Failures

Error Message

Cause

Diagnosis

Solution

Error Message

Cause

Diagnosis

Solution

CloudTrail Errors

Error Message

Cause

Solution

Observation

Possible Causes

Diagnosis

Solution

API Throttling

Error Message

Cause

Solution

Error Message

Cause

Solution

Empty or Missing Report

Observation

Cause

Diagnosis

Solution

Observation

Cause

Solution

Profile and Configuration Issues

Error Message

Cause

Diagnosis

Solution

Error Message

Cause

Solution

Script Execution Errors

Error Message

Cause

Solution

Observation

Cause

Diagnosis

Solution

Getting Help

Check AWS CloudTrail Logs

Enable Debug Logging

Community Support

AWS Support Resources

Quick Diagnostic Checklist