Streaming

story = ai "Write a short story" -> stream

for chunk in story {
    print_no_newline(chunk)  # incrementally
}

The call runs in the background and returns a stream. Consume it chunk by chunk, or use it as a value to block for the whole result:

full = string(story)

An ai stream is the same type as a file, a socket or an HTTP body - see std.io. It is the one stream that also has a value, which is what the line above uses; every other stream has to be read.

Untyped streams are real tokens

With a live provider, an untyped stream delivers the model's tokens as they arrive - genuine server-sent events, across OpenAI, OpenRouter and Ollama. One piece of code, three wire formats handled underneath.

Typed streams are not incremental

ai[T] "..." -> stream resolves through the normal coercion-and-retry path first and then chunks the completed result.

This is not a limitation to work around, it is the only correct behaviour: a schema cannot be validated against half an answer, and streaming a partial value that later fails coercion would mean emitting output you have to retract. If what you want is the appearance of progress on a typed call, that is what this gives you. If you want genuine incremental output, use an untyped stream.

Offline

Chunks are the mock text split into pieces. The consuming loop is exercised identically, so a streaming UI is testable with no key.

To a client

A stream pairs with a channel to push tokens out over HTTP as they arrive - see Streaming responses & SSE for the server side.

When the provider fails mid-stream

A provider can fail after tokens have already arrived - a rate limit, a content filter, an upstream outage - and reports it in a frame of its own. That frame raises an error rather than ending the stream, so a partial answer is never handed back as if it were complete. Catch it with try if a truncated result is better than none for your case.

Budget, cost and retries

A streamed call is a provider call, and is accounted for like one.

  • It counts against ECKO_AI_MAX_CALLS, so a loop that streams cannot spend past the cap you set.
  • It reports the tokens the provider used, so ECKO_TRACE records its usage and cost next to every other call.
  • It retries a transient failure - a timeout, a 429, an upstream 5xx - but only while nothing has reached you yet.

That last rule is the one worth knowing about. Once a chunk has been handed to your loop there is no taking it back, so retrying would deliver that chunk a second time and the text you assemble would be wrong with nothing downstream able to notice. A stream that fails after it has started raises instead, which is the error described above. Partial output with an error beats plausible output that is quietly doubled.

Restrictions

Cannot combine with voting (no partial majority), tools, or sessions.