+972 52-548-6969
AWS Roles, AWS Lambda and eventual consistency
I’m doing some cloud work, and I am working based off the official documentation, trying to automate the creation of an AWS Lambda. In order to allow me to quickly iterate, I’m basically creating the entire thing from scratch each time.
I have the following code:
- aws iam create-role --role-name $AWS_ROLE --assume-role-policy-document file://trust-policy.json
- aws iam attach-role-policy --role-name $AWS_ROLE --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- aws lambda create-function --function-name $FUNC_NAME --zip-file fileb://lambda.zip --handler lambda_function.lambda_handler --runtime python3.8 --role $ARN_ROLE
So far, so good, and exactly like it shows in the docs. But if you’ll run it as a script, it will fail with:
An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda.
If I re-run the exact same command, however, it works properly.
There is this interesting command, which indicates that roles are using eventual consistency:
aws iam wait role-exists --role-name $AWS_ROLE
Except… that this doesn’t work. It looks like there is some additional delay between creating the role, validating that it was created and when it is actually available for Lambda to be used.
After looking around and feeling like a fool, I added a sleep for 10 seconds to the script, and the problem went away.
I’m posting this for posterity sake and in the hope that someone can tell me that there is a better way. For now, I think I need a shower.
Comments
Actually it takes up to 60 seconds for roles to populate and when concurrent writes happen the behaviour is pretty much undetermined. In AWS many infrastructure operations are eventually consistent up to 60 seconds. Usually it is much faster in practice but retrying with backoff is the best strategy
This is one reason I use Terraform instead of raw CLI. It has facilities inside it for accounting for the eventual consistency issues in the AWS services. You also say that you are creating it from scratch every time. Does that mean that when you're done you delete the role? Lambda has issues with reusing the same IAM role name if the actual role has been recreated under the covers, and so (using Terraform), I let TF dynamically generate the IAM role name).
Consider you like C#, you can also use Pulumi, which basically wraps around TF, but let's you define the infra-as-code in developer-based languages. Terraform has their own infra-as-C# in beta and AWS CDK is the AWS native equivalent (wraps around Cloudformation).
Daniel,
Those are usually things that you do once ever, so I guess that makes sense. My gripe is that this is after I explicitly waited for it to be created.
Richard,
This is for test code, basically. Showing students how the cloud works behind the scenes.
Comment preview