In the early days of Emptor, we used to keep multiple IAM users for ourselves, which eventually multiplied our responsibilities to secure a growing number of access keys. This is a common problem with multiple AWS accounts, where each engineer has an IAM user for every account they use.
While growing our business, we also focused on improving the security of our systems to ensure the safety of our clients’ data. Our primary goal was to enhance the security of our production environments, beginning with a thorough review of AWS security best practices. We noticed that most engineers often had more permissions than needed for their daily work and that access keys were often not rotated for many months.
There are several risks associated with not rotating keys, including possible harm to our production systems if the access keys of one of our engineers were compromised.
Thanks to tools like aws-vault, we’re now able to store these keys securely on local machines and ensure authentication security for all engineers, but this left us with an additional problem: even when AWS keys are stored securely, they must be rotated regularly, and neither AWS nor aws-vault provides an automated way of doing this. Therefore, it became important for us to find a way to get engineers to rotate their AWS keys on a regular basis. To accomplish this, during one of our daily stand-ups, we developed the following rules:
- Engineers must have only one IAM user and can assume IAM roles as needed to obtain different sets of permissions in different accounts.
- Engineers’ default permission set must only allow them to make changes to their own IAM user (changing password, creating new access keys, setting up an MFA device, etc).
- Engineers must not be allowed to assume IAM roles unless they set up an MFA device.
- Engineers must not have more than one pair of access keys enabled at any time.
- Engineers must rotate their access keys and AWS management console password every 90 days.
We were able to implement most of these conditions with the help of existing IAM features, except for the requirement to enforce a 90-day expiration policy for access keys. Since we had a small engineering team, our first solution to enforce this particular condition was to remind engineers during a few meetings every month. It worked well for several months but did not scale as the number of teams increased and more and more engineers joined Emptor. We then decided to automate this procedure by developing an internal tool we call AWS Monitor.
To build this system, the first thing we needed to do was to tag each IAM user with useful data, so that AWS Monitor could discern who or what is using its credentials, and notify the owner of impending expirations of their access keys. We began with the following tags, which have proved to be very useful:
- type: This tag helped us differentiate between IAM users for engineers (
type=engineer) and machines (type=machine). - slack-id: This tag helped us identify the Slack account of the engineer to whom that IAM user belongs, so that we can send them private notifications via our system.
- full-name: This optional tag helped us improve the user interface of the AWS console itself. Since the console doesn’t mention the name of the person who owns the IAM user, adding it to a tag makes it easier to search for a specific user, and identify account owners during an audit.
Once we were able to differentiate between IAM users, we created a Slack application to send private notification messages to engineers and built a serverless application that runs every day to check the age of the access keys for every IAM user across all of our AWS accounts. It then notifies the relevant engineer thrice on Slack, using their Slack ID we stored as a tag, once their access keys have reached the ages of 76 days, 81 days, and 86 days. On the 91st day, it automatically disables their access keys.
We also send the same messages to a Slack channel where the Infrastructure Team can monitor the expiration and rotation of access keys with consistency. This monitor system also notifies the team when a new pair of access keys are created, so that we can verify whether these access keys were actually created by one of the engineers at Emptor or an unauthorized user. Neat, right?
This system has enabled us to monitor all of our IAM users, encourage our engineers to practice frequent access key rotation, and efficiently audit the age and usage of all of our access keys. It’s a small tool that took us only a few days to build but offers us invaluable information regarding the most vulnerable aspect of AWS. Our next goal is to explore the AWS SSO service. We believe it may provide us a set of features that will allow us to stop using IAM users indefinitely.
If you enjoy reading about our experiences with AWS, don’t forget to subscribe to our blog.
Happy AWS-ing!
