checkpoint

Resumable long-running jobs over std.sql: batch a source through a handler with the cursor committed in the same transaction as the writes, and memoise named pipeline steps. A crashed job resumes where it stopped and applies nothing twice.

ecko get github.com/ecko-lang/checkpoint
import checkpoint

Pure computation: it declares no capabilities, so it cannot touch the network, the filesystem or the environment.

Version 0.58.0 - source - MIT.


each_batch(db, name, source, limit, handler)

Runs the job name to completion, one batch per transaction, resuming from the last committed cursor, and returns { batches, items, done, cursor } for this call.

Each round calls source(cursor, limit), which must return { items: [...], next: cursor } (the cursor is null on the first call), then handler(items, tx), then stores next: the handler's writes and the new cursor commit together or not at all. If the handler throws, that batch rolls back, the error re-raises, and the next call resumes at the same batch. The run ends when the source returns no items, or items with next: null (a last page); the job is then marked done and later calls return at once without calling the source. batches and items count this call's work only; status has the totals. Write through tx (the same handle as db) and do not open another transaction on it inside the handler.

errors: checkpoint/bad_source when the source returns the wrong shape; checkpoint/stuck_source when it returns items without moving its cursor; checkpoint/not_storable when its cursor would not survive JSON; checkpoint/name_in_use when name belongs to a step; checkpoint/concurrent_run when another run moved the cursor mid-batch; bug/bad_argument for an empty name, a non-function, or a limit that is not a positive Int; whatever the source or handler throws.

checkpoint.each_batch(db, "backfill", src, 1000, fn(batch, tx) copy(batch, tx))

step(db, name, f)

Runs one named step of a pipeline once and returns its result. The first successful run stores the result; later calls return the stored value without calling f.

f runs inside a transaction on db together with the write that records it, so writes f makes through db commit only if the step is recorded; if f throws, nothing is stored, its writes roll back and the error re-raises. The result must survive a JSON round trip unchanged (null, bool, int, float, string, and lists and maps of those); anything else is refused. Both calls return the decoded form, so the first run and a resumed run see the same value. f must not open its own transaction on db.

errors: checkpoint/not_storable when the result holds a decimal, record, secret, bytes or function; checkpoint/name_in_use when name belongs to a batch job; bug/bad_argument for an empty name or a non-function; whatever f throws.

report = checkpoint.step(db, "nightly/extract", fn() extract())

status(db, name)

The stored state of a job or step, or null if name has never run (or was reset).

Fields: name, kind ("batch" or "step"), done, cursor (the last committed cursor of a batch job), value (a step's stored result), batches and items (totals across every run), and started_at, updated_at, finished_at in milliseconds since the Unix epoch.

errors: bug/bad_argument when name is not a non-empty string.

checkpoint.status(db, "backfill").cursor   # 812000000

reset(db, name)

Forgets a job or step, so the next each_batch starts from the beginning and the next step runs its function again. Returns whether there was anything to forget. It does not undo the writes the job made; clear those yourself first if a rerun would duplicate them.

errors: bug/bad_argument when name is not a non-empty string.

checkpoint.reset(db, "nightly/extract")   # true

sql_source(db, table, key, where = null, columns = null)

A keyset-paginated source over table for each_batch: each batch is the next limit rows with key > cursor, ordered by key, and the cursor is the last row's key.

It never uses OFFSET, so each batch costs the same however far in the job is, and gaps in the key are harmless. key must be unique on its own - the table's single-column primary key or the only column of a non-partial unique index - and this is checked when the source is built, because keyset over a non-unique key silently skips rows. Column names match without regard to case, as in SQLite. Rows whose key is NULL are never visited. where narrows the rows with a sql { ... } block (its holes bind as parameters); columns is a list of column names, and the key is always included. Items are row maps. SQLite only: it reads the schema through pragma_table_info.

errors: checkpoint/no_table, checkpoint/no_column or checkpoint/key_not_unique when the table does not fit; bug/bad_argument when a name is not a plain identifier, where is not a sql block or columns is not a list.

src = checkpoint.sql_source(db, "gen1", "id", where: sql { status = {"settled"} })

jsonl_source(stream)

A source over a JSON-lines stream for each_batch: each batch is the next limit lines, decoded, and the cursor is the byte offset just past the batch's last line, so a job that stops resumes at the next line with io.seek instead of reading every line again.

Open the stream yourself - fs.open("events.jsonl") - so the file access is yours and this package stays free of capabilities; a restarted run opens a fresh stream and resumes from the stored cursor. Blank lines are skipped, and CRLF line endings work. Only a file stream can seek.

errors: parse/bad_line (with offset, the byte where the line starts) when a line is not JSON; bug/bad_argument when stream is not a stream.

events = checkpoint.jsonl_source(fs.open("events.jsonl"))