Multi-tenant AI features

Per-tenant keys, tokens() and cost() to price each request, and a JSONL trace of every call - so inference spend is attributable without a separate metering microservice.

The problem

SaaS products add summarisation, classification, or generation per customer account. Finance needs to know what each tenant cost you this month; support needs to replay what the model returned for a specific ticket. The usual split is application code plus a wrapper SDK plus a logging pipeline plus a spreadsheet - none of which agree on token counts.

Why Ecko

Built-in metering: tokens(text) counts a prompt or a reply, and cost(in, out, in_price, out_price) prices it at your provider's rates. Log both with your tenant id in the same process - no sidecar counting HTTP responses from the provider.
Trace per request: ECKO_AI_TRACE appends one JSON line per inference call. Ship traces to object storage partitioned by tenant, or attach the relevant lines to a support export.
Provider config per env: Point staging at mock mode and production at your vendor via environment variables. The feature code path is identical; only the backend changes.

In practice

feature.ecko

import std.log

fn summarise(tenant_id, text) {
    out = ai "Summarise for the user: {text}"
    n_in = tokens(text)
    n_out = tokens(out)
    usd = cost(n_in, n_out, 0.15, 0.6)  # your provider's rates, per 1M tokens
    log.info("ai", { tenant: tenant_id, tokens: n_in + n_out, usd: usd })
    out
}

print(summarise("acme", "Three invoices are overdue and one is disputed."))

Multi-tenant AI features

The problem

Why Ecko

In practice

Further reading