terraform-aws-security-hub-report

Architecture

The Step Functions state machine in security-hub/step_functions.tf is the spine. Everything else hangs off it.

flowchart LR
    EB[EventBridge schedule] --> SFN[Step Functions]
    SFN --> D[discover Lambda<br/>account list]
    D --> M
    M --> C1[collect Lambda<br/>account 1]
    M --> C2[collect Lambda<br/>account 2]
    M --> CN[collect Lambda<br/>account N]
    C1 & C2 & CN --> S3P[(S3 parts/*.json)]
    S3P --> A[aggregate Lambda]
    A --> XLSX[(S3 report .xlsx)]
    A --> N[Slack / Teams / SNS]

State machine

stateDiagram-v2
    [*] --> Discover
    Discover --> Collect: run.accounts
    state Collect {
        [*] --> CollectAccount
        CollectAccount --> [*]: ok / partial / error
        CollectAccount --> CollectCrashed: Lambda error
        CollectCrashed --> [*]
    }
    Collect --> Aggregate
    Aggregate --> [*]
    Discover --> NotifyFailure: States.ALL
    Collect --> NotifyFailure: States.ALL
    Aggregate --> NotifyFailure: States.ALL
    NotifyFailure --> RunFailed
    RunFailed --> [*]

• Discover invokes the discover Lambda with an empty payload and stores its result under $.run.
• Collect is an inline Map over $.run.accounts, max_concurrency items at a time. Each item invokes the collect Lambda with account_id and parts_prefix. A collect invocation that crashes outright (as opposed to recording an error in its part file) is caught and replaced with a crashed marker so the Map continues.
• Aggregate invokes the aggregate Lambda with run_date, parts_prefix and accounts.
• Every task retries on Lambda service and throttling errors (3 attempts, 5s backoff, x2). Any other failure lands in NotifyFailure, which calls the aggregate Lambda with only an error key so it sends a failure notification, then the execution ends in RunFailed.

Lambdas

The three functions share one zip and one environment variable set (local.lambda_environment in lambda.tf) but have separate IAM roles (lambda_iam.tf). They run on python3.13 / arm64.

Function	Timeout	Memory	Role permissions
discover	60s	256 MB	organizations:ListAccounts
collect	900s	512 MB	sts:AssumeRole on SecurityHubReadOnlyCrossAccount in any account, S3 put under the report prefix
aggregate	900s	1024 MB	S3 get/put under the report prefix, SNS publish

discover (files/discover.py)

Returns {run_date, parts_prefix, accounts}. Accounts come from the ACCOUNT_IDS env var, or from organizations:ListAccounts (ACTIVE only, sorted) when it is empty. The parts_prefix includes the run time so two runs on the same day never share part files.

collect (files/collect.py)

One invocation per account. It assumes SecurityHubReadOnlyCrossAccount in the target account, then for every region in REPORT_REGIONS pages through GetFindings filtered to RecordState = ACTIVE and WorkflowStatus in (NEW, NOTIFIED), flattening each finding to one row. It writes <parts_prefix>/<account>.json and never raises: failures are recorded in the part’s status (ok, partial, error) and errors list. Security Hub calls use adaptive retry with up to 10 attempts to absorb throttling across many parallel accounts.

aggregate (files/aggregate.py)

Reads one part per discovered account (a missing part becomes a missing status), sorts all rows by severity, and writes a write-only openpyxl workbook with Summary and Findings sheets (bold light-blue headers, frozen header row, autofilter). It uploads the workbook and calls notify. If the event contains error (the state machine’s Catch output) it only sends a failure notification.

notify (files/notify.py)

One summary, three channels. A webhook on hooks.slack.com gets a Slack payload; any other webhook gets a Teams Adaptive Card. SNS gets the plain-text version. At most 20 failed accounts are listed in the message.

Contracts that span files

• HEADERS in common.py is the row layout written by collect, read by aggregate, and indexed by tests (row[5] is Severity). Change all of them together.
• The Step Functions payload keys (account_id, parts_prefix, run_date, accounts, error) must match the handler event keys.
• Environment variable names in lambda.tf must match the env() / env_list() calls in the Python.
• CROSS_ACCOUNT_ROLE in common.py, assume_role_arns in lambda_iam.tf, and RoleName in cloudformation/reporting-account-role.yaml are the same string.

Cross-account access

Nothing in Terraform touches other accounts. The module outputs collector_role_arn; every reporting account must have a SecurityHubReadOnlyCrossAccount role that trusts it, created by cloudformation/reporting-account-role.yaml (single stack or StackSet, see Deployment). The role carries the AWS-managed AWSSecurityHubReadOnlyAccess policy.

The bucket_policy and sns_topic_policy outputs are rendered from the real role ARNs so the output bucket and an external SNS topic can live in any account.

Design decisions

• Per-account fan-out instead of one long Lambda. Work per account is bounded by the 15 minute Lambda limit, not the whole run, so account count is limited only by patience: 300 accounts at concurrency 10 takes roughly 30 minutes.
• A failed account never fails the run. collect records the error in its part file and aggregate reports it in the Summary sheet and the notification.
• Part files are kept. They make a failed account’s error visible without re-running. Add an S3 lifecycle rule on SecurityHubReports/*/parts/ if you want them expired.
• Cost over resilience. No VPC, X-Ray, dead-letter queues, KMS on log groups, code signing or reserved concurrency. SNS uses the AWS-managed key. These show up as Checkov and trivy findings and are deliberate.
• Webhook URLs are sensitive. Plans redact the whole Lambda environment block as a result; that is expected.