IAM has really made me aware of my limitations as a run of the mill engineer and has me worried that it's exposed the boundaries of how far I can really scale. I keep telling myself that if I could simply get a couple days, uninterrupted, to focus on it, take it in, reflect, that I could become comfortable with it. But it never happens.
That said, this article cleared up a little bit of permission boundaries. It's now clearer to me that my organization is doing what the article speaks of, with cross-accounts and permission boundaries, and having the two-layer-admin clarified has not made my understanding worse. Good article.
That has been my experience as well. I don't understand it well enough to say that it's more complicated than it needs to be, but it's certainly clear that there is no obvious on-ramp. Everything starts locked down, and opening up anything requires figuring out which of numerous options gives you what you need without inviting the entire universe in.
Usually, the best way to learn is by experimentation. But with security, your "experiments" might leave you exposed. Or at least, leave your system in some nonstandard state, making it even harder to figure out what the correct route was supposed to be.
It's why I've been happy to use things like netlify. They may not scale, but at least it feels like I can get a project up without it becoming an ordeal of opening permissions.
It's not my main line of work, and as you say, I imagine I would figure it out if I did it consistently for a bit. But every time I start a side project using AWS, IAM scares me off.
I would suggest azure has an approach that shows how iam can be a lot simpler. I can create a resource group, and make you a contributor on that group.
You now have full permissions to servers, storage and anything else created under that group without me having to enumerate them. If I want to create a new blob store under that group, you already have access without me having to edit iam. I can still make granular permissions where suitable.
Azure makes the path to having all console users require mfa much simpler too.
AWS IAM does have resource condition keys which allow matching resources by account, org, org path, and tag. Resources in a Cloudformation stack would be automatically tagged, for example.
I think you're being a little harsh on yourself, but I can emphatise.
Every time I need to do something IAM related that's beyond the obvious, I find that I have forgotten everything I learned and thought I hammered into my head the last time...
I find reading AWS documentation on the subject almost unbearably tedious, so I need to go and re-read tutorials and re-watch videos. Coincidentally, this is what I am doing again this morning!
If you're in the kind of megacorp where you have no choice to implement this crap because of your size then good luck staring into the void but the actual realistic advice for everyone else is.
1. Specify your infrastructure as code, doesn't matter what tool; can be bash for all it matters. But no manual changes to your AWS account(s).
2. Monorepo your company's app(s) and the IaC together. Obviously allow them to be deployed independently but make it so it's not a huge PITA to make a code change with a corresponding infra change in a single PR.
3. Require reviews from someone with infra experience to check MRs that change infra and use that as an opportunity to narrow permissions.
If there are so many infra changes that your ops team is overwhelmed reviewing PRs then your permissions are too narrow.
Everything you said is wise advice but has nothing to do with whether or not, or how, to use perm boundaries. IAM delegation and role/policy creation is independent of whether it's created manually or through Terraform.
I could be the one missing something but I’m saying to not use permission boundaries at all until you’re at a scale where it’s unavoidable due to the complexity of your corporate structure.
The thing rolling out changes to your infra once merged is god. Boundaries don’t do anything in this model because “asking for the permission boundary to be changed so I can make X change” is the same as “ask to make X change.”
Both require approval of the same people.
Once you graduate to polytheism is when you need boundaries.
You can only have a single permissions boundary attached to your role that is limited to 6144 characters, which is fairly small. Use AWS accounts as your boundary instead.
This would be great advice, but until AWS have a more solid story about resource sharing (to avoid duplicated costs for your usual boilerplate account infra) then you’ll quite quickly end up amalgamating accounts to reduce costs again, in my experience.
RAM is getting there, but I tried this pattern again recently and ran headfirst into the problem of not being able to easily share a R53 zone between accounts without it descending into a mess of assume-role.
I’m sold on the simplicity of small team accounts rather than larger accounts with complicated IAM policy though.
Perm boundaries have their place, but yes, in my previous role, each department/app group was allocated their own AWS account in the organization. Maybe we didn't get as fragmented as you recommend, because you could still have similar but different groups of people within an account working on projects and permission boundaries were still used.
regarding the frustrations with size limits: I feel the same about so many of the constraints in AWS IAM. Also on my gripe list: Using raw JSON to specify policy rather than a better DSL for expressing complex logic. Trying to get certain conditions properly specified can be a mind bender with the syntax provided, or be downright impossible.
We use both. Have a managed project where people can create their own IAM roles using defined policies for all services. For some use cases creating your own role is better so they must attach one of the roles they create with the project as their boundary when they create the role.
There are no constraints on the allowed contents of an AWS IAM policy, and that’s too much power to give an author. I wish it were a capability-based system: whatever privileges I may have, I can pass on some or all of those, but no more.
IAM has really made me aware of my limitations as a run of the mill engineer and has me worried that it's exposed the boundaries of how far I can really scale. I keep telling myself that if I could simply get a couple days, uninterrupted, to focus on it, take it in, reflect, that I could become comfortable with it. But it never happens.
This diagram was pretty critical in helping me "get it": https://docs.aws.amazon.com/IAM/latest/UserGuide/images/Poli...
Thank you for taking the time to link this. I appreciate it, this is indeed very helpful.
That said, this article cleared up a little bit of permission boundaries. It's now clearer to me that my organization is doing what the article speaks of, with cross-accounts and permission boundaries, and having the two-layer-admin clarified has not made my understanding worse. Good article.
That has been my experience as well. I don't understand it well enough to say that it's more complicated than it needs to be, but it's certainly clear that there is no obvious on-ramp. Everything starts locked down, and opening up anything requires figuring out which of numerous options gives you what you need without inviting the entire universe in.
Usually, the best way to learn is by experimentation. But with security, your "experiments" might leave you exposed. Or at least, leave your system in some nonstandard state, making it even harder to figure out what the correct route was supposed to be.
It's why I've been happy to use things like netlify. They may not scale, but at least it feels like I can get a project up without it becoming an ordeal of opening permissions.
It's not my main line of work, and as you say, I imagine I would figure it out if I did it consistently for a bit. But every time I start a side project using AWS, IAM scares me off.
I would suggest azure has an approach that shows how iam can be a lot simpler. I can create a resource group, and make you a contributor on that group.
You now have full permissions to servers, storage and anything else created under that group without me having to enumerate them. If I want to create a new blob store under that group, you already have access without me having to edit iam. I can still make granular permissions where suitable.
Azure makes the path to having all console users require mfa much simpler too.
Thanks for sharing this perspective.
AWS IAM does have resource condition keys which allow matching resources by account, org, org path, and tag. Resources in a Cloudformation stack would be automatically tagged, for example.
https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p...
I think you're being a little harsh on yourself, but I can emphatise.
Every time I need to do something IAM related that's beyond the obvious, I find that I have forgotten everything I learned and thought I hammered into my head the last time...
I find reading AWS documentation on the subject almost unbearably tedious, so I need to go and re-read tutorials and re-watch videos. Coincidentally, this is what I am doing again this morning!
If you're in the kind of megacorp where you have no choice to implement this crap because of your size then good luck staring into the void but the actual realistic advice for everyone else is.
1. Specify your infrastructure as code, doesn't matter what tool; can be bash for all it matters. But no manual changes to your AWS account(s).
2. Monorepo your company's app(s) and the IaC together. Obviously allow them to be deployed independently but make it so it's not a huge PITA to make a code change with a corresponding infra change in a single PR.
3. Require reviews from someone with infra experience to check MRs that change infra and use that as an opportunity to narrow permissions.
If there are so many infra changes that your ops team is overwhelmed reviewing PRs then your permissions are too narrow.
Agreed with everything you said! IaC changes everything. Especially CDK.
But IMO permission boundaries still have a place in certain cases.
If you allow SREs to login and poke around in the accounts, what will stop them from creating policies in ad hoc way?
Yes you can make RO roles but these don’t always work the way they are advertised.
But I totally agree with your approach otherwise and that’s a winner setup for us as well on several projects.
Everything you said is wise advice but has nothing to do with whether or not, or how, to use perm boundaries. IAM delegation and role/policy creation is independent of whether it's created manually or through Terraform.
I could be the one missing something but I’m saying to not use permission boundaries at all until you’re at a scale where it’s unavoidable due to the complexity of your corporate structure.
The thing rolling out changes to your infra once merged is god. Boundaries don’t do anything in this model because “asking for the permission boundary to be changed so I can make X change” is the same as “ask to make X change.” Both require approval of the same people.
Once you graduate to polytheism is when you need boundaries.
You can only have a single permissions boundary attached to your role that is limited to 6144 characters, which is fairly small. Use AWS accounts as your boundary instead.
This would be great advice, but until AWS have a more solid story about resource sharing (to avoid duplicated costs for your usual boilerplate account infra) then you’ll quite quickly end up amalgamating accounts to reduce costs again, in my experience.
RAM is getting there, but I tried this pattern again recently and ran headfirst into the problem of not being able to easily share a R53 zone between accounts without it descending into a mess of assume-role.
I’m sold on the simplicity of small team accounts rather than larger accounts with complicated IAM policy though.
Why not give each account and region its own zone?
Perm boundaries have their place, but yes, in my previous role, each department/app group was allocated their own AWS account in the organization. Maybe we didn't get as fragmented as you recommend, because you could still have similar but different groups of people within an account working on projects and permission boundaries were still used.
regarding the frustrations with size limits: I feel the same about so many of the constraints in AWS IAM. Also on my gripe list: Using raw JSON to specify policy rather than a better DSL for expressing complex logic. Trying to get certain conditions properly specified can be a mind bender with the syntax provided, or be downright impossible.
We use both. Have a managed project where people can create their own IAM roles using defined policies for all services. For some use cases creating your own role is better so they must attach one of the roles they create with the project as their boundary when they create the role.
I place AWS Permission Boundaries in the "Write once, read never" category
There are no constraints on the allowed contents of an AWS IAM policy, and that’s too much power to give an author. I wish it were a capability-based system: whatever privileges I may have, I can pass on some or all of those, but no more.
> whatever privileges I may have, I can pass on some or all of those, but no more.
That's just a permission boundary in AWS.
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_poli...
You can create constraints as the ones you describe using Organizations and SCPs(Service Control Policy)
Create a system to manage IAM roles.