No Country for Old Console Clicks: Slack-Driven IAM Identity Center Access

What I Built

The client runs a multi-account AWS Org with about thirty accounts and about a dozen IAM Identity Center permission sets. Engineers get access by joining IC groups, and every group grants a specific permission set on a specific account or set of accounts. Before this bot, access requests were a ticket to Platform Engineering, and PE clicked around in the IC console to add people to groups. The lag was fine when there were a few requests a week. It got noisy when the org grew.

Goal was to move that flow into Slack with two-eyes approval baked in. Requester types a slash command, picks a user and a group from a modal, submits. The bot posts an approval message to a private PE-only channel. Any PE member (except the requester themselves) can click Approve or Reject. On approve, the bot calls Identity Store APIs directly to add the membership. On reject, the bot posts a short close-the-loop message in the same channel and stops.

Every action leaves an audit trail in the approval channel: who asked, who approved, what group, what user, when. CloudTrail already records the actual IC API calls, but the Slack channel is the human-readable log.

The Four Slash Commands

Command	What it does
/ic-request	Add an existing IC user to a group. Two-eyes required.
/ic-revoke	Remove an existing IC user from a group. Two-eyes required.
/ic-onboard	Create a new IC user, add them to a group, generate a one-time password, and DM the OTP to the approver so they can hand it off in person. Two-eyes required.
/ic-groups	List every requestable group and what each one grants (account IDs and permission set name). No approval, ephemeral response, self-serve.

The commands all live on the same Lambda. Slack routes each one to a shared API Gateway endpoint, and the handler dispatches based on the command name in the payload.

The Terraform-MR Backend I Threw Away

My first design was Terraform-native. The bot would open a merge request in the repo that manages IC, the MR would add the new membership as an aws_identitystore_group_membership resource, PE would review the MR, the CI pipeline would apply, and the user would get access when the pipeline finished. On paper this looked clean: audit trail via Git, no direct API calls, no runtime privilege on the bot's IAM role beyond opening MRs.

Two things killed it once I actually built the first version.

The org's Terraform convention manages permission sets, groups, and group-to-account assignments in code. It does not manage user memberships in code, because those churn too fast (someone joins a project for two weeks, leaves, comes back for a hotfix, leaves again). Every membership change would have been a commit on the main branch of the IC repo, and the diff noise would have swallowed anything real. When I ran the first end-to-end test, the MR passed review but the plan showed dozens of unrelated drift entries because the state hadn't been refreshed in weeks and the CI pipeline surfaced every stale value. That's not the bot's fault. That's a signal the pattern is wrong for this domain.

The second thing: the pipeline felt slow for a Slack-driven flow. From MR open to membership active was about fifteen minutes on a good run. Engineers who type a slash command expect a Slack-shaped response time, not a CI-pipeline-shaped one. Fifteen minutes is fine for a one-off change. It is terrible when someone is trying to unblock themselves on a Friday afternoon.

I scrapped it and rewrote the bot to call IC APIs directly. The Lambda has a scoped IAM role with identitystore:CreateGroupMembership, identitystore:DeleteGroupMembership, identitystore:CreateUser, and read permissions to describe groups and users. Grants take about two seconds end-to-end. The audit trail lives in CloudTrail (for the AWS side) and the Slack channel (for the human side), which is where PE was already looking.

Slack's Three-Second Ack Window

Slack requires every interaction endpoint (slash command, view submission, block action) to return HTTP 200 within three seconds. Miss it and Slack shows the user a red banner that says "the command didn't work." No retry, no queue, just a bad UX and a puzzled engineer wondering if their access request went through.

Three seconds is not a lot of time. My real work per approval was: verify the Slack HMAC signature (fast), look up the requester and target in IC (about 300ms per call), call CreateGroupMembership (about 500ms), post two Slack messages to update the channel and DM the requester (about 300ms each), log to CloudWatch. Warm invocation, add it up and I was at four or five seconds. Cold invocation with Lambda init included, seven or eight. Slack never saw the ack.

The pattern that fixed it is what I ended up calling async self-invoke. The Lambda has two entry points on the same handler, distinguished by a marker on the event payload:

def lambda_handler(event, context):
    if event.get('_source') == 'ic-access-bot-worker':
        return handle_async_work(event)
    return handle_sync_slack_request(event, context)

The front handler verifies the Slack signature, extracts the relevant IDs, and re-invokes the same Lambda function via lambda:InvokeFunction with InvocationType='Event' (async). It stamps the marker on the outgoing payload so the second invocation knows to run the async path. Then it returns 200 to Slack in under 200 milliseconds. The second invocation, running in parallel, does the actual IC API calls and posts the resulting Slack messages using the standard Web API.

This is a well-known Slack integration pattern but is worth naming clearly: don't try to do real work inside a slash-command handler. Ack first, work second, post the result out-of-band.

Cold Start Optimization

Even with async self-invoke, one thing still had to fit inside three seconds: opening the modal when the user types /ic-request. Modals need Slack's fresh trigger_id, which is only valid for a few seconds and can't be handed off to an async invocation. That path had to stay synchronous.

Warm invocation was fine, about 400ms end-to-end. Cold start was the problem: Lambda init took about 1.4 seconds (imports plus boto3 clients), then the sync work took another 1.5 seconds. That put cold-start modal opens at about 2.9 seconds, right at the edge. Slack's actual timeout is more like 2.7 seconds in practice. Users were seeing intermittent failures the first time they used the bot each morning.

Fix was to move as much work as possible from the request path into the Lambda init path. Init is billed but it happens outside the Slack ack window. Specifically, I moved the SSM Parameter Store fetches for the Slack signing secret and bot token into module-level code:

import boto3
_ssm = boto3.client('ssm')
SLACK_SIGNING_SECRET = _ssm.get_parameter(Name='/ic-bot/slack-signing-secret', WithDecryption=True)['Parameter']['Value']
SLACK_BOT_TOKEN = _ssm.get_parameter(Name='/ic-bot/slack-bot-token', WithDecryption=True)['Parameter']['Value']

Those two SSM calls are about 200ms each. Moving them from request to init cut warm-path latency by 400ms and had no effect on cold-start (they were happening anyway, just later). Modal opens now consistently return in about 1.7 seconds cold, well inside the window.

The SCP Was the Actual Guardrail

Deployed the bot, granted the role IAM permissions I thought were sufficient, ran the first end-to-end test. Got:

AccessDeniedException: User: arn:aws:sts::111111111111:assumed-role/ic-access-bot-role/... is not authorized to perform: identitystore:CreateGroupMembership

The IAM policy on the role explicitly allowed identitystore:CreateGroupMembership on *. IAM Access Analyzer said it was fine. So did the IAM simulator. The API call kept failing.

The culprit was an SCP on the Security OU called IC-Delegated-Admin-Protection. Someone had written it to deny identitystore:CreateGroupMembership and identitystore:DeleteGroupMembership from every principal except an ArnNotLike whitelist of four specific role ARNs. The idea was that only IC delegated admins could mutate memberships, even from within the security account. My bot's role wasn't on the whitelist.

Adding the role to the SCP's whitelist fixed it. And having gone through the debug, I found the design of that SCP worth thinking about. IAM permission is one layer. The SCP is the actual load-bearing kill switch: even if someone accidentally attached AdministratorAccess to a role in that OU, they still couldn't add themselves to any IC group. The bot's role now lives inside the whitelist as a documented exception. If the SCP were removed, we'd have a much softer guardrail. That was a lesson worth learning by hitting it live.

Test Mode for Solo Smoke Testing

The bot enforces requester != approver. Fine in production. Rough when you're one person trying to run end-to-end smoke tests, because every test requires roping in a second PE member to click Approve. I could have asked the tech lead every time, but every ping meant startling him with a fake @-mention out of context.

Added an IC_BOT_TEST_MODE environment variable. When set to 1:

• The requester != approver guard is bypassed. I can click Approve on my own requests.
• The "Approved by" line in the channel post is formatted as [TEST] approver:U0123456 instead of the usual <@U0123456>. That's the string that would otherwise render as a real @-mention. In test mode it's just plain text, so nobody gets notified.

Turning test mode on and off is one CLI call:

aws lambda update-function-configuration \
    --function-name ic-access-bot \
    --environment "Variables={IC_BOT_TEST_MODE=1}" \
    --region us-east-1

Cheap to add, saved me from pinging people multiple times during smoke tests, and made it easy to re-verify the bot after any code change without needing a second human in the loop. I disable it before each real deploy.

/ic-groups Async via response_url

The /ic-groups command lists every requestable group and what each one grants. Sounded easy. Turned out to be the second time I hit the three-second wall.

The org has about thirty requestable groups. To describe what each grants, I have to list all groups, and for each group call sso:ListAccountAssignmentsForPrincipal to find every (account, permission set) pair it grants. Best case that's thirty API calls in sequence. Worst case, with a few groups that grant assignments across many accounts, it's ninety. Even warm that ran four to eight seconds. No way it fits in three.

I couldn't do the async self-invoke pattern here because /ic-groups doesn't open a modal, it just posts a response. But Slack has an escape hatch for exactly this: when you get a slash command, the payload includes a response_url that stays valid for thirty minutes. You can POST to that URL at any point in the next thirty minutes with a message body, and Slack will render it as if the slash command had returned it.

So the flow is:

1. Front handler receives the slash command, extracts response_url.
2. Front handler async-invokes the worker with the response_url in the payload.
3. Front handler immediately returns a placeholder ephemeral message: Fetching group catalog, one sec...
4. Worker iterates the groups, computes the listing, and POSTs the full message to the response_url.
5. Slack replaces the placeholder with the real content in-place.

User experience is a brief spinner-like flow instead of a red timeout banner. Same async self-invoke pattern I already had. The only new part was reading and re-using response_url. Slack's docs are good on this if you want the details.

What Surprised Me

• How much of the work was in the corners. The happy-path grant flow was maybe a day of code. Getting the async pattern right, hunting down the SCP denial, tuning cold start, adding test mode, and building the response_url pattern for the helper command took the rest of the week.
• Direct API calls read cleaner than I expected. I was ready to defend the MR-backed design as "the right way." Direct API turned out to be simpler, faster, and easier to reason about, once I let go of the idea that everything has to live in Terraform.
• The SCP was doing more work than the IAM policy. If you're building anything that touches IC in a mature org, look at the SCPs on the Security OU first. They probably say something about IC.
• Slack's response_url is the escape hatch that saves you. When something legitimately takes more than three seconds, response_url lets you keep the slash-command UX without ripping the whole architecture apart.

Cost

Under two dollars a month.

• Lambda: about a hundred invocations per weekday, most under 500ms of billed duration. Free tier covers it.
• API Gateway: same story, well inside free tier.
• SSM Parameter Store: standard parameters, free.
• CloudWatch Logs: less than 50MB per month at INFO level.

The main cost was the four days of engineering. Compared to what the PE team was previously spending on click-ops-to-grant-access, it paid back inside a month.

Want Help With This?

If you're building something similar (Slack-driven IC access, an internal ChatOps bot, or any pattern that has to fit inside Slack's three-second ack window), reach out via the contact form. Happy to talk through the tradeoffs.