Why I Didn't Just Use Celery
A job queue is a promise that work you accepted will eventually happen. Making that promise honestly is the hard part.
What It Takes to Build a Job Queue
10 posts
- Part 1
- Part 2
Two Stores, One Job, and the Gap Between Them
Splitting durable truth from fast dispatch buys a lot. It also creates a dual write I never made atomic.
- Part 3
Three Priority Tiers in One Number
A single Redis sorted set gives strict priority and FIFO inside each tier. Then I checked the arithmetic.
- Part 4
Retries, Backoff, and Why Jitter Is Not Optional
Retrying is easy. Deciding when to retry, and making sure everyone doesn't decide the same thing, is the actual work.
- Part 5
Idempotency Is a Race Until the Database Says Otherwise
Check-then-insert looks correct and isn't. The only reliable fix is to stop checking and let a constraint decide.
- Part 6
A Timeout Doesn't Stop the Work
Deadlines protect the worker, not the job. Once blocking code is involved, the difference stops being academic.
- Part 7
What 'Processing' Means When the Worker Is Gone
A status column records what a job was doing. It cannot record whether anyone is still doing it.
- Part 8
The Load Test That Was Supposed to Fail
173,000 requests with zero errors told me nothing useful. The test that broke its own thresholds told me everything.
- Part 9
Tracing a Job That Outlives Its Request
Parent and child is a claim about containment. Queued work isn't contained by anything, so it needs a different relationship.
- Part 10
The Reliability Envelope I Didn't Finish
An honest accounting of what this job queue guarantees, what it doesn't, and the config settings that do nothing at all.