ecko test

ecko test                  # discover and run everything
ecko test tests/           # an explicit path
ecko test --jobs 4         # run up to four files at once
ecko test --filter parse   # only the cases with "parse" in their name
ecko test --no-capture     # print what tests print as they run
ecko test --generate file.ecko   # draft tests for a file with AI

Discovery

Two conventions, both honoured:

• anything under tests/, at any depth
• *_test.ecko anywhere in the tree

Both walk the whole tree, so a suite split across subdirectories is found. Four kinds of directory are skipped: vendor/, because a dependency's tests belong to its maintainer and would fail for reasons you cannot fix; target/ and node_modules/, which are build output; and anything beginning with a dot. Symlinked directories are not followed.

A package keeps its tests in tests/. The release workflow's archive step takes only the top-level *.ecko files, so tests/ stays out of the published zip, which is what you want; a root-level <name>_test.ecko would ship with it.

Writing a test

import std.test

test.case("splits on commas", fn() {
    test.eq(split("a,b", ","), ["a", "b"])
})

test.case("rejects an empty name", fn() {
    test.err(fn() validate(""))
})

test.eq, test.ok, test.err and test.fail are the assertions. See std.test.

Each file runs in its own process

State in one test file - module-level variables, a port a fake server bound, os.pid() - never reaches another. Files run one after another by default and their output streams as they go. --jobs N (or -j N) runs up to N files at once; each file's output is buffered and printed in file order, so the report reads the same. Use it when files do not share a port, a database or a fixture directory.

Running some of the cases

ecko test --filter parse
ecko test tests/parse_test.ecko --filter=header

--filter TEXT runs only the cases whose name contains TEXT, or whose test.group has a name that contains it, in which case every case in that group runs. The match is a plain substring and is case-sensitive. It combines with paths and --jobs.

A case that does not match is not called, printed or counted. A file with no matching case prints nothing, not even its path, unless the file itself fails (an error outside every case), which is always shown. A run that matches no case at all fails with no test case matches 'TEXT', so a typo in the filter cannot pass green.

Mock mode is forced

ecko test ignores your API key. Every ai call in a test run answers from the deterministic mock, whether or not a provider is configured.

This is not a convenience, it is the point. It means a test suite over an AI pipeline is fast, free, offline, and identical on every machine - so the suite can run on every commit, and a failure is a real regression rather than a model having a different day. What mock mode cannot assert is answer quality; that is what contracts are for, and they are evaluated in tests too.

Output and exit status

Each file's path, one line per case under it, a summary at the end, and a non-zero exit if anything failed:

tests/parse_test.ecko
  ok splits on commas
  ok rejects an empty name
1 file(s), 2 case(s): 2 passed, 0 failed (0.01s)

A failing case reads FAIL <name> - <why>, followed by where it failed in the test file and, for a test.eq over lists or multi-line strings, a line diff (- expected, + got):

tests/a_test.ecko
  FAIL totals - expected 2, got 1
      at tests/a_test.ecko:6
  FAIL rows - expected [1, 2, 3], got [1, 2, 4]
      at tests/a_test.ecko:9
        1
        2
      - 3
      + 4
1 file(s), 2 case(s): 0 passed, 2 failed (0.01s)

The at line is the innermost call in the test file: the test.eq line, or, when the assertion is in a helper in another file, the line that called into it. A group's N of M cases failed line has none.

What a case prints is held back and shown only under its failure, behind | , so it is attached to the case that printed it:

  FAIL parses the header - expected "v2", got "v1"
      at tests/parse_test.ecko:12
      output:
      | reading header.txt

A passing or skipped case's output is dropped. Each case keeps at most 64 KiB; past that the report says how many more bytes there were. Output printed outside every case (setup at the top of a file), and output from tasks a case spawned or pmap workers, prints as it happens. --no-capture turns holding back off, for when you are debugging a test that passes. Running a test file directly (ecko tests/parse_test.ecko) never holds output back.

A case that called test.skip reads SKIP <name> - <reason>, and the summary counts it apart: 3 case(s): 2 passed, 0 failed, 1 skipped. Skips do not change the exit status. The summary mentions skips only when there are some.

--generate

ecko test --generate parser.ecko

Reads the file and drafts a test file for it. It is a starting point, not a substitute for deciding what the behaviour should be - read what it writes before trusting it. It is one of the few commands that genuinely needs a key: with no provider configured it says so and exits 1, rather than writing mock tests.