Most recently, CVE-2021-44228 pushed all of us into adopting slapdash measures to protect our applications. One technique that gained traction was the WAF, or Web Application Firewall. A WAF protects your web applications or APIs against common web exploits that may affect availability, compromise security, or consume excessive resources. AWS supports this via the AWS WAF offering, and gives payer accounts the flexibility to deploy it centrally across child accounts using AWS Firewall Manager. My experience with both offerings is limited, but a single deployment cycle turned up a handful of gotchas that are either hidden in plain sight or missing from the docs entirely. I hope this helps other folks in security or infrastructure make more informed deployments.
18KB payloads
If you look at the AWS docs for WAF, you'll find a red box with this message:
Only the first 8 KB (8,192 bytes) of the request body are forwarded to AWS WAF for inspection. For information about how to manage this, see Web request body inspection.
I was puzzled the first time I read this: all someone needs to do is pad a malicious payload with 8KB of data and AWS WAF will let it through. This blog post demonstrates the problem well. AWS's fix is to block any request with more than 8KB of payload data, which means if your application needs to accept 8KB or more at any point, you're out of luck. Most of us read the docs before deploying an AWS service, but this gotcha is exactly the kind that can quietly break an application in production.
2Replacement discrepancy
When using AWS Firewall Manager to centrally deploy WAF WebACLs across child accounts, you'll see the following options in the policy configuration stage.
If you have an ALB or API Gateway resource running without a WAF WebACL in a child account, selecting the auto-remediate option will automatically move that resource behind the WebACL this policy creates. The gotcha is in this line:
Replace web ACLs that are currently associated with in-scope resources with the web ACLs created by this policy
In theory, leaving this unchecked means resources behind an existing independent WAF WebACL stay untouched by the Firewall Manager policy. That holds true only if the independent WebACLs are running WAFv2. If they're still on the deprecated WAF Classic, leaving the box unchecked does nothing. I found this out the hard way: a batch of applications behind WAF Classic got moved to the new WAFv2 policy despite the box being unchecked. Most of those WAF Classic instances had custom rules and rate limits, none of which carried over, and their protection broke.
When I raised this with AWS support, they initially denied it. After they reproduced it themselves, they accepted the bug and suggested using tags to exclude resources from future Firewall Manager policies. Unsurprisingly, none of this is documented anywhere on their site.
3Migration caveats
Firewall Manager will quietly migrate your resources from WAF Classic to WAFv2 (see above), but there's no easy way to do that migration yourself. AWS does have a page walking through migration via a CloudFormation template, with a few caveats attached. That template, or any other migration path, will not carry over your custom rules, managed rules, rate limit rules, or logging, as the docs note. Makes me wonder if it'd be faster to just spin up your own Terraform or CloudFormation than rely on AWS's half-assed migration steps.
4False positives
We've run AWS WAF on a number of services over the past few months. We also use the WAF offerings from Fastly, Akamai, and Cloudflare on other projects, and AWS WAF has a noticeably higher false positive rate than any of them. A Web Application Firewall is not a "live and let live" kinda tool. You have to constantly monitor false positives and false negatives and tweak the rules to get the protection you actually want, and AWS WAF is no exception.
5Bad parsing
We hit a strange bug in production where legitimate multi-part requests with XML data were getting blocked by the WAF. After investigating and confirming with AWS Support, it turns out there's a parsing bug in AWS WAF that, as of this writing, still hasn't been fixed. A compressed XML payload without line breaks gets blocked, but the same payload with line breaks added sails through the WebACL. Definitely one of those head-scratching issues.
6Extra pennies per month
This gotcha isn't a big one, but it belongs in the "uncomfortable" column. Firewall Manager creates WebACLs in child accounts regardless of whether those accounts have any resources (ALB, API Gateway, etc.) or the replacement box is unchecked. A WebACL costs $5 a month, and a single rule inside it adds another $1. Say you have 10 accounts that don't need a WAF yet but might later, so you've added them as child accounts to the Firewall Manager policy just in case. Until that day comes, you're paying 10 × $6 = $60 extra every month. Probably pennies for a company running 100+ accounts that don't need a WAF, but I wanted to put it out there. Thx.