#cloudtrail — Public Fediverse posts on home.social

Tamás @[email protected] · 2025-03-05 · 15:27 UTC

I wanted to automatically disable an IAM user when it does something suspicious. Since this IAM user is used by a script I know that when it deviates that is a good indicator that it was compromised and I need to investigate.

How hard could it be?

Well, it turned out to be a frustrating experience. CloudTrail records events done by users, so this should be easy to setup. But then I started to encounter problems:

* Only the first CloudTrail event is free, so I did not want to create more than one trail
* CloudTrail sends events to EventBridge but only for the current region, which is not enough

I have an organizational trail in the management account. Let's see how easy it is to send these events into a member account!

* CloudTrail can send events to a CloudWatch log then I could set up a subscription filter. This worked for a PoC but ultimately there is a limit of 2 for subscription filters for a log group. So this was a no-go
* Otherwise it writes to S3, so I had to have a Lambda reading the objects as CloudTrail writes them

At this point I had a Lambda that got all CloudTrail events and filters out the interesting ones: ones with AccessDenied error, GetCallerIdentity, and ConsoleLogin. That should be a good start.

EventBus Rules can send events based on a filter, so forwarding these events into an EventBus seems like a good idea. So so far the chain is: CloudTrail => S3 => Lambda => EventBus.

But how can I send these events to the member account? Well, an EventBus Rule, of course. So I created an EventBus in the member account.

Next issue: a CloudFormation stack can't create an EventBus Rule in a different region. Interestingly, it is possible to create *cross-account* but not cross-region. So I needed an EventBus in the target region as well and set up a Rule to forward events there.

Then the very last step is to set up a Rule to filter events for the IAM user(s) and set up a CloudWatch alarm that calls a Lambda that attaches the DenyAll policy to the user.

Since I wanted everything managed by CloudFormation I ended up with an enormous amount of stacks:

* (mgmt acc us-east-1) CloudTrail + Lambda + EventBus
* (mgmt acc us-east-1) EventBus Rule to forward events to the member account
* (member acc us-east-1) EventBus to receive events from the mgmt account
* (member acc us-east-1) EventBus Rule to forward events to the regional EventBus
* (member acc eu-west-1) The target stack with the IAM users and an EventBus to receive events to

What makes it a particularly annoying experience is that there are so many small limitations that make a simpler solution impossible:

* CloudTrail should support filtering by events so that the whole management account => member account part could be saved
* Or: the default EventBus should receive *all* CloudTrail events not just ones for the current region
* EventBus Rule should be allowed to be cross-region. That would have saved me one EventBus
* EventBridge Pipes don't support SNS as a source and also it's not clear if that supports cross-region and cross-account pipes

I wrote about my frustrations in this article: [https://advancedweb.hu/cloudtrails-horrible-developer-experience/](https://advancedweb.hu/cloudtrails-horrible-developer-experience/).

Overall, I'm fairly happy with this solution, but I feel that it would be so much easier if AWS supported some basic features around CloudTrail.

#aws #cloudtrail #eventbridge

Originally published [on my blog](https://advancedweb.hu/shorts/how-hard-it-is-to-disable-an-iam-user-when-it-does-something-suspicious/)

#aws #cloudtrail #eventbridge