In part one of this blog series, “Serverless Querying and Evaluating of Logs Using Athena,” we discussed the importance of logging and monitoring in your organization and how to implement it in a serverless way for the most critical AWS services. We also showed you how to use partition projection in Athena to fully automate partition management. These partition projections are configured through Glue parameters, with each table having an account parameter that includes all AWS Account IDs required as partitions to query for.
However, the question arises of how to keep these parameters automated and up-to-date with the current status of the AWS Organization. New accounts can be created at any time, or accounts can join the AWS Organization via invitation. To avoid manually maintaining all parameters of the Glue tables for these events, we have an automation solution for you.
ℹ️ We actually decided to not remove AWS Account Ids from the partition parameter whenever an AWS Account leaves the AWS Organizations or will be SUSPENDED, because the logs are still valid and maybe needed whenever an request for this account from the past is made in the future.
Architecture Overview
Our automation creates an EventBridge where the Management Account of our AWS Organization has access to perform a “PutEvent” action. This PutEvent action to our central EventBridge will be performed each time a new AWS Account is successfully created in our AWS Organization (CreateAccountResult) or an existing AWS Account accepts the AWS Organization invitation (AcceptHandshake). This event will trigger a Lambda function through our central EventBridge, which will update our existing Glue tables for the new AWS Account ID for the specific Glue table parameter required as input for the AWS Accounts partition. After this automated update, you will be able to query your logs immediately after your new AWS Accounts have been created or your AWS Account has joined your AWS Organization.
Open Source
We have published the complete code of the solution freely on Github as we want to give something back to the community. We hope you find this solution helpful in automating your indexes for Glue / Athena tables! If you have any feedback about the solution, please feel free to reach out to us or open a github issue.
Conclusion
Automating the Glue partitions will help you with your daily work so that no one needs to worry about adding new AWS Accounts to the partition parameters. This ensures that decisions due to missing information in log queries are prevented, and your queries for the information sources are kept up to date.
