--- title: "Your LLM spend cap is bypassable: X-Forwarded-For vs CloudFront-Viewer-Address, and why you also need an origin lock" published: false description: "Three quiet bugs in a per-IP spend cap on Lambda behind CloudFront and API Gateway, the fix for each, and a live demo you can curl." tags: aws, serverless, security, webdev canonical_url: https://liftstack.gaperi.ro/guides/spend-cap-bypass --- *If you put a per-user or per-IP free-tier cap in front of a paid call on Lambda behind CloudFront and API Gateway, the obvious key is spoofable, the execute-api URL is a side door, and an in-memory counter overshoots. Here is each bug and its fix.* ## The setup You run a Lambda behind CloudFront and API Gateway. In front of a paid call — an LLM completion, say — you put a free tier: so many calls per anonymous user per day, then you stop. The counter needs a key. The obvious choices are the caller's IP or a user id the client sends. Both are a mistake, and the mistake is quiet: nothing errors, the bill just grows. This is three bugs in that design and the fix for each, ending with commands that exercise a live demo. The key function and the origin lock are copied from code running in production; the atomic counter in Fix 3 is newer and runs in the demo. ## Bug 1: keying on a header the caller controls The naive version keys the counter on `X-Forwarded-For`, or on an `x-user-id` the client supplies. It usually looks like this. ```ts // DON'T: the client controls this value. function callerIp(event) { const xff = event.headers['x-forwarded-for'] ?? ''; return xff.split(',')[0].trim(); // left-most entry } ``` *The spoofable key.* CloudFront is both the reason this fails and the tell. When a viewer request already carries `X-Forwarded-For`, CloudFront appends the viewer's real IP to the end of the header before it reaches your origin, leaving whatever the client sent to the left. Reading the left-most entry hands you an attacker-chosen value. The right-most entry is no better: API Gateway appends the CloudFront server's address after it. Don't parse this header at all. So the cap is free to defeat. Rotate the header and every request looks like a fresh IP, so a new bucket is minted on each call. ```bash for i in $(seq 1 100); do curl -s -X POST https://your-app.example/api/llm \ -H "X-Forwarded-For: 10.0.0.$i" done # every request looks like a new IP, so the per-IP cap never trips ``` *A hundred paid calls, one real client, against any API keyed this way.* > An `x-user-id` you let the client send is the same bug without the CloudFront detail: the client picks the key, so the client picks the bucket. ## Fix 1: key on CloudFront-Viewer-Address Key on `CloudFront-Viewer-Address`. CloudFront derives it from the viewer's connection; it holds the IP and the source port, for example `198.51.100.10:46532`. When your origin request policy adds it, CloudFront overwrites any value the viewer sent, so the client cannot choose it. Two conditions attach. First, the header has to reach your Lambda: it is a viewer-location header, which you forward with an origin request policy (it is not available in a cache policy). The managed `AllViewerExceptHostHeader` policy — the one this site's `/api/*` behaviour uses — forwards all viewer-location headers and drops `Host`, which is what API Gateway and Lambda function URL origins need. Confirm the header actually arrives at your handler, and fail closed if it is missing. Second, strip the port and treat a missing value as no-key, not as a free pass. This is the whole key function from the code behind fiscura.ro. ```ts export function ipKey(event: APIGatewayProxyEventV2): string | null { const viewer = event.headers['cloudfront-viewer-address']; if (viewer) return `ip:${viewer.replace(/:\d+$/, '')}`; // strip the port if (process.env.NODE_ENV !== 'production') return 'ip:dev'; return null; } ``` *Fails closed only if you set NODE_ENV=production; Lambda does not set it for you.* > Header names arrive lowercased in the HTTP API v2 payload, so read `cloudfront-viewer-address`, not `CloudFront-Viewer-Address`. > A per-address key has one more gap: IPv6. CDK turns IPv6 on for CloudFront distributions by default, and one host can hold many addresses (on AWS, one network interface can take a /80 prefix), each with its own bucket and no forged header needed. Key IPv6 viewers on their /64 prefix or wider. The demo below does; the function above does not yet. ## Bug 2: the execute-api URL is a side door There is a second entrance. The `execute-api` URL API Gateway gives your HTTP API is public by default: anyone who has it can call it directly. You can disable that default endpoint, but then the API answers only on a custom domain name, which is just as reachable from the internet, so turning it off moves the side door rather than closing it. A direct caller skips CloudFront entirely, so it can send any `CloudFront-Viewer-Address` it wants. That means Fix 1 only holds for traffic that genuinely came through your distribution. Straight to `execute-api`, the viewer-address header is caller-controlled again, and you are back to Bug 1. ```bash curl -s -X POST https://332dsafb8d.execute-api.eu-central-1.amazonaws.com/demo/cap \ -H 'CloudFront-Viewer-Address: 203.0.113.99:443' # without an origin check, the origin trusts this ``` *The side door. This site's API is locked, so here the same command gets a 401 (see Try it).* > The Lambda event's `sourceIp` will not save you here: it is the IP of the immediate TCP connection to API Gateway, which for CloudFront traffic is the edge node, not the viewer. Allow-listing CloudFront's IP ranges admits every distribution, not just yours. ## Fix 2: an origin lock, and why it answers 401 Lock the API to CloudFront with a secret only CloudFront can present. Configure a custom origin header — call it `X-Origin-Verify` — with a long random value on the API origin. CloudFront adds it to every request it sends there and overwrites any same-named header a viewer sends; a caller going straight to `execute-api` can send the header too, but does not know the value. AWS documents this pattern for keeping callers from bypassing CloudFront: HTTPS on both legs, and a value you rotate periodically, accepting old and new while the change rolls out. The Lambda compares the header against the expected secret in constant time and refuses anything else. `timingSafeEqual` throws if the two buffers differ in length, so compare lengths first. ```ts function viaCloudFront(event: APIGatewayProxyEventV2): boolean { const expected = process.env.ORIGIN_VERIFY; if (expected === undefined) return true; if (expected === '') return false; const sent = event.headers?.['x-origin-verify']; if (typeof sent !== 'string') return false; const a = Buffer.from(sent, 'utf8'); const b = Buffer.from(expected, 'utf8'); // Constant-time, so response timing cannot guess the token byte by byte. // timingSafeEqual throws on buffers of different lengths, so a length // mismatch is answered first; that reveals only the length, which is no // secret (the stack fixes it at 40 characters). return a.length === b.length && timingSafeEqual(a, b); } ``` *Empty secret = misconfigured, refuse. Unset = local runs and tests, so it fails open: make sure every deployed stack sets it.* The refusal is 401, not 403, on purpose. A direct `execute-api` call never passes through CloudFront, so its refusal reaches the caller as sent. But a request that did come through CloudFront can still fail the check (mid-rotation, say), and CloudFront custom error responses apply to the whole distribution, not per behaviour. This site maps 403 to its `/404.html` page, so a 403 from the API would reach the client as that HTML page, with a 404 status. CloudFront can remap only 400, 403, 404, 405, 414, 416 and 500–504, so the API refuses with 401 and caps with 429, which always pass through. This site leaves 404 alone, so the API's own 404s pass through too. > The header lock is the simplest lever for an HTTP API, which cannot use AWS WAF, resource policies or private endpoints (REST API features); a REST API can also check the header in a WAF rule. Alternatives: CloudFront origin mutual TLS to an HTTP API custom domain that requires it, with the default endpoint disabled; or a Lambda function URL behind CloudFront origin access control (AuthType AWS_IAM, and POST/PUT clients must send the body's SHA-256 in x-amz-content-sha256). ## Bug 3: counters that overshoot With the right key and the side door shut, the counter itself can still overshoot, in two ways. First, an in-memory counter. On standard Lambda, each concurrent request gets its own execution environment, which handles one request at a time. A `Map` in module scope is therefore per container: with N warm containers your effective cap is at least N times what you set, and each count resets whenever Lambda replaces a container, which it does every few hours even under steady traffic. Second, even a shared store overshoots if you read, then decide, then write. Check-then-act leaves a gap. ```ts // DON'T: two steps with a gap between them. const used = await countFor(key); // read if (used >= CAP) return capped(); await model.invoke(prompt); // paid call await increment(key); // write, after the fact ``` *Under a burst, many requests read "under the cap" before any writes land.* They all pass the check and all make the paid call, so the cap overshoots by however many requests arrive in that gap. ## Fix 3: one atomic conditional UpdateItem Collapse read, check and increment into a single conditional `UpdateItem`. DynamoDB applies writes to an item in the order received, and evaluates the condition against the latest version of the item, so concurrent calls queue on the item rather than racing each other. ```ts await ddb.send(new UpdateItemCommand({ TableName: table, Key: { k: { S: key } }, UpdateExpression: 'ADD #n :one SET expiresAt = if_not_exists(expiresAt, :ttl)', ConditionExpression: 'attribute_not_exists(#n) OR #n < :cap', ExpressionAttributeNames: { '#n': 'n' }, ExpressionAttributeValues: { ':one': { N: '1' }, ':cap': { N: String(cap) }, ':ttl': { N: String(ttl) }, }, ReturnValues: 'UPDATED_NEW', })); // ConditionalCheckFailedException => already at the cap, return 429 ``` *The expression strings are exactly those in the demo's consume() (api/src/lib/caps.ts).* `ADD` on a number adds to the current value, or creates the attribute at that value when the item is new; `if_not_exists` sets the TTL on the first call only. When the count is already at the cap the condition is false, DynamoDB throws `ConditionalCheckFailedException`, and the handler returns 429. No burst gets past the cap. Three limits remain. The update is not idempotent: if it lands but the response is lost, the AWS SDK can retry it and count that call twice, so the cap errs toward refusing early. It counts calls, not dollars: bound each call's cost (output tokens, input size), or reserve the worst case before the call and settle after. And it bounds each address, not the total: many real addresses mean many buckets, so keep one global daily ceiling behind it. > The demo stores the address only as an HMAC-SHA256 hash, with a TTL so DynamoDB deletes the row. The reset comes from the UTC date in the key, not from TTL: TTL deletes within a few days of expiry, and an expired row can still be read until then. ## Try it The guide page has a live widget backed by `POST https://liftstack.gaperi.ro/api/demo/cap`: ten calls per viewer IP (IPv6: per /64) per UTC day, keyed on `CloudFront-Viewer-Address`, refused with 429 after that and reset at 00:00 UTC. Three commands show the fixes. ```bash # 1) a forged X-Forwarded-For is ignored; your real viewer IP is counted curl -s -X POST https://liftstack.gaperi.ro/api/demo/cap -H 'X-Forwarded-For: 203.0.113.99' # 2) a forged CloudFront-Viewer-Address through CloudFront is overwritten curl -s -X POST https://liftstack.gaperi.ro/api/demo/cap -H 'CloudFront-Viewer-Address: 203.0.113.99:443' # 3) straight to execute-api, past CloudFront: the origin lock refuses it curl -s -X POST https://332dsafb8d.execute-api.eu-central-1.amazonaws.com/demo/cap -H 'CloudFront-Viewer-Address: 203.0.113.99:443' ``` *The first two count your real address; the third never reaches the counter.* The first returns 200 (429 with `capped: true` after ten calls in a UTC day), with `countedAs` set to your IP as CloudFront saw it, port stripped (for IPv6, its /64 prefix). Under `ignored` it echoes the `X-Forwarded-For` the Lambda received: your forged value, your real IP (appended by CloudFront), then a CloudFront server's address (appended by API Gateway). That is Bug 1, live. The second counts your real IP too: CloudFront replaced the forged header. The third returns 401 `{"error":"not_via_cloudfront"}`. A 429 without `capped` is API Gateway's rate limit, not the cap. A version of this lock runs in production behind fiscura.ro, where it answers 403: a direct call with a forged viewer address is refused, while the identical request routed through the site returns 200. The live demo with buttons is on the original page: https://liftstack.gaperi.ro/guides/spend-cap-bypass#try-it *The demo stores a keyed hash of your IP address (for IPv6, of its /64) with today’s count, never the address. DynamoDB deletes it after midnight UTC, usually within a day or two.* ## Checklist - Key the cap on CloudFront-Viewer-Address, port stripped, IPv6 on its /64 — never X-Forwarded-For or a client-supplied id. - Forward that header with an origin request policy, and fail closed when it is missing. - Lock the API to CloudFront with a secret origin header, compared in constant time; check that every deployed stack sets it. - Refuse with a status CloudFront cannot remap (401, 429), never one your distribution remaps (here, 403). - Count with one conditional UpdateItem, not an in-memory counter or a read-then-write. - Bound each call's cost, and keep one global daily ceiling behind the per-address cap. - Store a keyed hash of the IP, not the IP, with a TTL. ## Disclosure > Disclosure: I build Liftstack, a serverless-SaaS starter kit for AWS. It is in development and not for sale yet. The cap key above is from the code behind fiscura.ro, and the constant-time origin lock from keepvista.com, both of which I run; I am packaging them, with the atomic counter, into the kit. > This article was written with the help of AI and fact-checked before publishing.