outbox
Transactional outbox over std.sql: write your business row and the message you owe in the same transaction, then relay it with at-least-once delivery, retry and backoff.
ecko get github.com/ecko-lang/outbox
import outbox
Pure computation: it declares no capabilities, so it cannot touch the network, the filesystem or the environment.
Version 0.56.0 - source - MIT.
init(db)
Creates the outbox table and its relay index if they do not already exist, and returns null. Safe to call every time your program starts.
errors: sql when the table cannot be created.
outbox.init(db)
add(db, topic, payload, id = null, max_attempts = 5, now = null)
Queues a message and returns its id.
Call this inside the same sql.transaction as your business write - the row you are recording and the message you owe both commit together, or neither does. payload must be plain data that JSON gives back unchanged (null, bools, ints, floats, strings, and lists and maps of those), so deliver_fn receives exactly what was added.
id is your own id (default a fresh uuid.v7()); max_attempts is how many delivery attempts relay makes before dead-lettering this message; now is the Unix ms to stamp it with (default time.now() - inject a fixed value in tests).
errors: outbox/not_storable when payload holds a decimal, record, secret, bytes or function; outbox/duplicate_id when a message with id already exists; bug/bad_argument for an empty topic, a non-string id, a max_attempts that is not a positive Int or a non-Int now; sql when the write fails.
outbox.add(tx, "order.created", { order_id: "o-1" }, max_attempts: 10)
relay(db, deliver_fn, batch_size = 100, base_backoff_ms = 1000, max_backoff_ms = 60000, now = null)
Delivers the pending messages that are due, oldest first, and returns { delivered, failed, dead } counts.
Calls deliver_fn(msg) for each - msg has id, topic, payload, status, attempts, max_attempts, created_at, next_attempt_at, delivered_at and last_error. deliver_fn's return value is ignored; it signals failure by throwing.
A message that returns normally is marked delivered. A message that throws has its attempt recorded and is rescheduled with exponential backoff (base_backoff_ms * 2^(attempts - 1), capped at max_backoff_ms) until it reaches its max_attempts, when it is marked dead instead. Each deliver_fn call is caught on its own, so one bad message is recorded and skipped rather than blocking the batch. At most batch_size messages are read per call; now (Unix ms, default time.now()) picks which are due.
Delivery is at-least-once: a crash between a successful deliver_fn call and the update that marks it delivered means the message is retried on the next relay, so dedupe on msg.id. Run one relay at a time per table: two concurrent relays read the same pending rows and both deliver them.
errors: bug/bad_argument when deliver_fn is not a function, a count or delay is not a positive (or, for base_backoff_ms, non-negative) Int, or now is not an Int; sql when the table cannot be read or updated. deliver_fn's own errors are never re-thrown.
outbox.relay(db, fn(msg) publish(msg), batch_size: 50) # {delivered: 3, failed: 0, dead: 0}
pending(db)
The messages awaiting delivery, oldest first.
Includes messages that already failed once and are backing off, not only ones never attempted - compare next_attempt_at if you need to tell them apart.
errors: sql when the table cannot be read.
len(outbox.pending(db)) # 3
dead(db)
The messages that reached max_attempts and will not be retried automatically, oldest first. Use retry_dead to give one another chance.
errors: sql when the table cannot be read.
map(outbox.dead(db), fn(m) m.id) # ["msg-2"]
retry_dead(db, id, now = null)
Moves a dead message back to pending with its attempt count reset, so the next relay picks it up at now (Unix ms, default time.now()). Returns null.
errors: outbox/not_found (with id) when id names no dead message; bug/bad_argument when now is not an Int.
outbox.retry_dead(db, "msg-2")
prune(db, older_than)
Deletes delivered and dead messages last touched before older_than (Unix ms) and returns how many rows it removed. Pending messages are never pruned, however old - they are still waiting on a real outcome.
errors: bug/bad_argument when older_than is not an Int; sql when the delete fails.
outbox.prune(db, time.now() - 7 * 86400000) # 120