Designing a Flash-Sale Seat Reservation System in AWS (Part 2): Never Sell a Seat Twice

TL;DR: Every booking goes through one Redis Lua script. The script checks, in order, whether the candidate is already admitted, whether their level is full, and whether any seat is left overall. Then it counts the seat and marks the candidate. All of this happens as one atomic step (nothing else can run in between). Lua won over MULTI/WATCH (too many retries when many requests compete) and distributed locks (extra round trips while holding the lock). An in-memory counter can still be wrong in four ways. Each needed its own fix:

• Double rollbacks: the undo step has a guard, so running it twice does nothing the second time.
• Lost replies: every call carries a nonce (a random value that identifies one request), so a retried call gets its original slot back.
• Crashed requests: a list of pending admissions, plus a cleanup job, returns seats that never reached the database.
• Failover: a booking counts only after the Redis replica confirms it has the write (WAIT 1 200). Otherwise the booking is undone and the client retries.

This is Part 2 of a three-part series. Part 1 explains why seat admission lives in Redis rather than a database. Part 3 covers what happens after a seat is admitted.

The invariants

What exactly does “never sell a seat twice” mean?

Before writing any code, we wrote down three invariants (rules that must always be true). Everything in this post exists to keep them true.

1. Caps hold. The number of admitted seats per level never exceeds that level’s cap, and the total never exceeds the global cap.
2. One live booking per candidate. A candidate code is linked to at most one seat that is held or paid.
3. No leaks. Every admitted seat becomes either paid or released in the end. No seat stays counted with nobody holding it.

Invariants 1 and 2 are what “zero double bookings” means. Invariant 3 stops the event from ending with unsold seats that the system forgot about.

The state

What lives in Redis?

Key	Type	Purpose
booking:open	string	The admin’s open/close switch
candidates:valid	set	Allowlist of eligible candidate codes, loaded before opening
candidate:email:<code>	string	The email on file for each code
adm:seen	set	Candidate codes that currently hold a seat
adm:total	integer	Seats admitted overall
adm:level:<L>	integer	Seats admitted per level
adm:pending	hash	code → nonce, level, time, slot for admissions not yet safely in the database
hold:*	strings with a TTL (they expire on their own)	Per-payment stashes (temporary copies): the booking record, the payment URL, and the execute and rollback guards

Two configuration choices matter more than they seem to:

• maxmemory-policy noeviction. When memory runs low, Redis must refuse writes with an error. It must never silently delete adm:seen to free space. Losing that set would allow duplicates. A write error, on the other hand, simply becomes a 503 that the client can retry.
• Normalised identifiers. Candidate codes have their surrounding spaces removed and are converted to uppercase before any lookup. Otherwise the counter would treat ab123 and AB123 as two different people. A SHA-256 hash of the normalised code is the database key. Its first characters become the queue message group and the payment’s invoice number.

Gates, then the script

Why check some things outside the script?

The gates are simple read checks. They reject most invalid traffic before any write happens:

1. Is booking open? If not, 503 closed.
2. Is the code on the allowlist? If not, 400 invalid_code. The allowlist is loaded into Redis from the back-office database before opening. There is deliberately no fallback to the database while the system runs. A code that is missing from the allowlist must not turn into a database query at 50,000 requests a minute.
3. Does the email match the code? If not, 400 email_mismatch.

A hidden honeypot field is checked before all of these. (A honeypot is a form field that real users never see, so only bots fill it in.) A form that fills it in gets a fake success response that looks real, with no admission and no log entry.

The gates can live outside the script because their values change only when an operator changes them, before the event. The checks that compete with each other at the moment of booking are all inside the script:

-- KEYS: adm:seen, adm:total, adm:pending
-- ARGV: code, max_total, level, level_cap, nonce, now_ms
local code, level, nonce = ARGV[1], ARGV[3], ARGV[5]
local level_key = 'adm:level:' .. level

-- 1. Already admitted? A retry of the same request gets its slot back.
if redis.call('SISMEMBER', KEYS[1], code) == 1 then
  local p = redis.call('HGET', KEYS[3], code)
  if p then
    local n, lvl, _, slot = string.match(p, '^([^|]*)|([^|]*)|([^|]*)|([^|]*)$')
    if n == nonce then return {tonumber(slot), lvl} end
  end
  return {-1, ''}                                  -- duplicate
end

-- 2. Level full?
local cap = tonumber(ARGV[4])
if cap > 0 and tonumber(redis.call('GET', level_key) or 0) >= cap then
  return {-3, level}                               -- level full
end

-- 3. Any seat left overall?
if tonumber(redis.call('GET', KEYS[2]) or 0) >= tonumber(ARGV[2]) then
  return {-2, ''}                                  -- closed
end

-- 4. Admit: count, mark, and record as pending.
local slot = redis.call('INCR', KEYS[2])
redis.call('INCR', level_key)
redis.call('SADD', KEYS[1], code)
redis.call('HSET', KEYS[3], code, nonce .. '|' .. level .. '|' .. ARGV[6] .. '|' .. slot)
return {slot, level}

The order matters. The duplicate check comes first, so a candidate who already has a seat is told so, even after their level fills up. The level check comes before the global check, so the user learns the most specific reason. Only the last branch writes anything. A rejection therefore costs a few reads on one thread.

The API loads the script once with SCRIPT LOAD and calls it with EVALSHA. After a restart or failover, the new primary has an empty script cache and answers NOSCRIPT. The client reloads the script and retries once.

Why not the other atomic building blocks in Redis?

Option	Problem under a flash sale
INCR, then check, then DECR if the counter went too far	The counter briefly goes above the cap, and readers can see that. The duplicate check is a separate step, and another request can run between the steps, so two requests for the same candidate can both pass.
MULTI/WATCH (optimistic transactions)	Every booking at the same level watches the same key. In the opening second, nearly all transactions abort and retry. That creates a storm of retries on the busiest key.
Distributed lock (for example Redlock) around a read, change, write sequence	At least two extra round trips while holding the lock, and a lock service that is one more thing that can fail.
Lua script (chosen)	Runs from start to finish on the Redis thread. There is no lock to hold across the network and no retry loop.

Two more design details keep the rest of the system simple:

• “Level full” is published once. When a level fills, a SETNX latch (a key that only the first writer can set) makes exactly one API instance rewrite status.json. Thousands of requests cannot compete to publish the same change.
• A duplicate can continue. Suppose a candidate’s booking is still HELD and they submit again, perhaps because their first request timed out in the browser. The 409 response then carries their existing payment URL. The URL is read from a Redis stash (never from the database), and the page redirects them straight to payment. Without this, a user whose first response was lost could not reach the seat they already hold.

Failure mode 1: undo that runs twice

What happens when rollback runs twice?

A seat must be given back when a booking fails: the payment could not be created, the enqueue failed, or the hold expired without payment. Several paths can trigger that release. Early in load testing, two of them sometimes ran for the same booking, and the counters went negative. The undo step decreased the counters every time it ran, even though removing the candidate from the set only had an effect the first time.

The fix was to make undo a script that only acts if the candidate is still admitted:

if redis.call('SREM', KEYS[1], ARGV[1]) == 0 then return 0 end  -- already undone
redis.call('DECR', KEYS[2])                                     -- total
redis.call('DECR', 'adm:level:' .. ARGV[2])                     -- level
redis.call('HDEL', KEYS[3], ARGV[1])                            -- pending entry
return 1

That fixes repeats, but not a stale (out of date) undo. Suppose a candidate’s first payment is released and they book again. Then a late event for the first payment arrives, such as a TTL expiry or a repeated callback. A guard that only asks “is the candidate admitted?” passes, because the candidate is admitted again. The stale event then frees the new seat.

So every release that a payment triggers goes through a second guard, keyed by payment ID. SET hold:rollback:<paymentId> NX EX 86400 and the undo run in the same script. A payment can release a seat at most once, and only the payment that owns the seat can release it.

The rule we took from this: any operation that more than one path can trigger must be idempotent, and the guard that makes it idempotent must live in the same atomic step as the action itself. (Idempotent means it is safe to run more than once, with the same result.) A check followed by a separate action is a race.

Failure mode 2: the reply that never arrives

What if the script succeeds but the API never hears about it?

Networks drop replies. When that happens, the Redis client does the sensible thing and retries the command. For the admission script, that sensible retry is a disaster.

The first call admitted the candidate. The retry finds them in adm:seen and returns “duplicate”. The user sees “already registered” with no payment link, and their seat stays counted, with no hold behind it.

The fix is a nonce for each request. The API generates a random nonce for each booking request and passes it to the script. On admission, the script records it in the adm:pending hash (step 4 above). When a call finds the candidate already admitted, it compares nonces. The same nonce means this is the same request, retried, so the script returns the original slot. A different nonce means a real second attempt, so it returns duplicate.

We also run the admission with a context that is separate from the HTTP request’s cancellation. If a user closes the tab during the request, that cannot stop the call halfway through its retry.

Failure mode 3: the request that dies halfway

What if the API crashes between admitting and enqueueing?

The hot path is three steps: admit, create payment, enqueue. If the process dies after the first step, the seat is counted, but no record exists anywhere that could expire it. Neither the database nor the queue knows the seat exists. That would break invariant 3.

The pending ledger closes the gap. (The ledger here is simply the list of admissions that are not yet safe in the database.) Every admission writes an adm:pending entry in the same atomic step. /book clears the entry only after the enqueue has succeeded, because from that point on the queued record carries the booking. Any rollback clears the entry atomically too.

On every sweep, the reconciler (the single background runner described in Part 3) looks for pending entries older than 15 minutes:

• If a database record exists for that candidate, the booking completed. The reaper (the cleanup step) removes the entry.
• If no record exists, the request died between steps. The reaper undoes the admission, with the same nonce check, so it can never undo a newer attempt.

Fifteen minutes is longer than the 10-minute hold plus the queue’s worst-case delay. A slow but healthy request is never cleaned up by mistake.

Failure mode 4: the failover that forgets

Can a Redis failover sell a seat twice?

Yes, if you let it. ElastiCache replication is asynchronous. The primary confirms a write to the client before the replica has received it. If the primary dies in that gap, the replica is promoted to primary without the last few admissions.

At the last seat, that gap means one seat sold twice. The same applies to a candidate: the promoted replica does not know they booked, so they can book a second seat.

We compared three options.

Option	Trade-off
Accept the risk	The window is milliseconds long and failover is rare. But it breaks the one promise the client cared about, at exactly the moment when load is highest.
Count seats in the database too	Correct, but it puts the database back on the hot path, which is what Part 1 worked to avoid.
Wait for a replica acknowledgement (chosen)	Send the script and WAIT 1 200 together in one pipeline: wait up to 200 ms for one replica to confirm it has the write. If the replica confirms, the admission counts. If not, undo it and return a 503 that the client can retry.

Under normal conditions, replication lag between Availability Zones is a few milliseconds, so the wait adds almost nothing to a normal booking. Both commands go out in one pipeline, so there is no extra round trip.

This choice has two consequences we accepted deliberately:

• Bookings fail closed. That means: if the replica is down, every booking returns 503 until ElastiCache replaces it. We chose correctness over availability for this one path. The page retries with increasing delays, and replacing a replica takes minutes, not hours.
• WAIT is not consensus. It makes a lost admission far less likely. It does not make Redis strongly consistent. With one replica, the node that confirmed the write is the node that gets promoted. That covers the realistic failure: the loss of one node or one Availability Zone.

As a final safety net, every five minutes the reconciler compares the Redis counters with a count of the records in DynamoDB. It raises an alert only if the mismatch is still there on two checks in a row, because records normally lag behind the counter by a few seconds.

How we tested it

How do you convince yourself a counter is right?

• Unit tests against an in-memory Redis for every branch of the script: caps, duplicates, nonce replay, idempotent and stale undo, and the pending reaper.
• Race tests that send thousands of admissions at the same time against a small cap. They check that the admitted count equals the cap exactly and that no candidate appears twice.
• A real primary and replica in Docker for the replica acknowledgement path. In-memory fakes usually report zero replicas, so they cannot test it.
• Full-rate load tests with 40,000 test candidates against the real stack. They pass only if admissions stop exactly at the cap and the counters match the database afterwards.

On launch day the invariants held. 20,700 seats were sold, with zero double bookings and zero overbookings.

What’s next

An admitted seat is only half a booking. Part 3 follows it through payment. It covers why a payment redirect can confirm a booking but never release one, and how a single runner makes every release decision. It also covers how a rate-limited gateway token is shared across the fleet, and how the queue and DynamoDB keep up without ever slowing down a booking.