Engineering note
Durable queues: retries without the mystery
Retries are easy to add and hard to make honest. A look at leases, idempotency keys and dead-letter replay in the queue that powers background work on weond.
A background job that fails once is an incident. A background job that fails forever is a data problem. Most of the work in a queue system is not delivering messages — it is deciding what to do when delivery goes wrong.
weond queues are signed HTTP delivery queues. Producers enqueue a JSON payload, consumers receive a signed POST, and the platform tracks the result. That sounds simple until you ask three questions: who owns a message while it is being processed, how many times may it be delivered, and where does it go when nobody can process it?
Leases, not locks
When a consumer receives a message, the platform grants a lease. The message stays reserved for the duration of the lease and becomes visible again if the consumer does not acknowledge it in time.
This is deliberately not a lock. A crashed consumer should not block a queue forever, and a slow consumer should not be able to hold a message indefinitely without saying so. Consumers can extend a lease while they are still working, which makes long-running jobs explicit instead of accidental.
- Visibility timeout controls how long a message stays reserved.
- Heartbeat extension keeps a healthy long job alive.
- Max attempts decides when a message stops being retried.
Idempotency is the contract
At-least-once delivery means your handler will occasionally run twice. There is no clever queue configuration that removes this requirement; there is only a handler that can tolerate it.
Every delivery includes a stable message ID and an attempt number. We recommend writing the ID to your database inside the same transaction as the side effect:
insert into processed_messages (message_id)
values ($1)
on conflict do nothing;
If the insert reports zero rows, the work was already done. Acknowledge the message and move on.

Dead letters are a feature
When a message reaches its maximum attempts, it moves to the dead-letter queue. The payload, headers, attempt history and last response are preserved. From the dashboard you can inspect the last error and replay the message with a fresh attempt budget.
Replay is intentionally manual. Automatic replay loops hide bugs and create traffic storms. A human decision is cheap; a queue that retries a broken job for three days is not.
The goal is not zero failures. The goal is failures that are visible, bounded and reversible.
Operating a queue you can trust
The most useful metrics are not throughput. They are age of the oldest visible message, lease expiration rate and dead-letter growth. Those three numbers tell you whether the queue is keeping up, whether consumers are healthy and whether something is quietly broken.
You can create a queue from the project dashboard and inspect its deliveries through the CLI reference. Start with a small max-attempt count, make your handler idempotent, and let the dead-letter queue be boring.