ecko
Use casesData pipelines and migrations
A billion-row backfill
It runs once, against real records, and it has to finish. Enrichment fetched ahead of the transform kept the hot loop from blocking on lookups.
The problem
Every company that outlives its first data model eventually has to move to the second one, and the backfill is the frightening part. There is no rehearsal. It runs against real rows, usually on a deadline, and a job that dies at record nine hundred million leaves you asking which rows committed and whether running it again double-processes them.
This one moved 1.3 billion transaction records out of a first-generation channel integrator schema into its replacement. It was built in Python first. That version was a nightmare to get through a run and was abandoned; the rewrite went through the whole set without a single failure.
Why Ecko
- Enrichment that runs ahead of the row
- Related lookups are fetched concurrently for a whole batch before the transform needs any of them, so the loop never waits on a round trip. That is most of the difference between a job that finishes and a job you sit and watch.
- Arithmetic that will not be quietly wrong
- Integer overflow raises instead of wrapping around. On a money column the failure you want is the one that stops, not the one that keeps going with a smaller number.
- Nothing to install on the box
- One binary, and no garbage collector deciding to pause somewhere in hour four. Memory is freed at scope exit, so a long run holds what it is using and no more.
In practice
## Fetch every account for the batch up front, in parallel, so the
## transform below never waits on a round trip.
accounts = pmap(batch, fn(r) lookup(r.account_id))
for (row, account) in zip(batch, accounts) {
write(transform(row, account))
}
Try it on your workload.