A Check That Cannot Fail

I want to describe a bug I wrote, because it is the most instructive one I have produced in a year and because the category it belongs to is sitting in your practice right now.

I have a pre-publication check for this site. Its job is narrow: before anything goes live, confirm that no published article links to an article that is not published. A dead internal link is a small thing, but it is exactly the kind of small thing that should be mechanical rather than remembered.

I wrote it as a shell loop. Roughly this:

CLEARED="alpha beta gamma"
for slug in $CLEARED; do
  grep -qs "draft: false" "posts/$slug.md" \
    || echo "PROBLEM: $slug is not published"
done && echo "Safe to publish."

Run it, get “Safe to publish,” push. It said that every time.

It had never checked anything.

What actually happened

Two independent defects, either of which alone would have been enough.

Defect one: zsh does not word-split unquoted parameter expansions.

This is the default macOS shell, and it is not bash. In bash, $CLEARED in a for list undergoes word splitting on IFS and yields three words. In zsh it does not. It yields one word — the entire string.

I set up a directory with alpha.md and beta.md published and gamma.md marked draft: true, then ran the identical script under both shells. Actual output:

### zsh
PROBLEM: alpha beta gamma is not published
Safe to publish.
exit=0

### bash
PROBLEM: gamma is not published
Safe to publish.
exit=0

Under zsh the loop body executed once, against a file named posts/alpha beta gamma.md, which does not exist. grep returned non-zero because it could not open the file, not because the content was wrong. The || branch fired and printed one line naming a concatenated nonsense slug — a line that looks enough like a real per-item message to scroll past unread, especially in a script whose output you have learned prints something harmless.

gamma — the actual problem — was never examined. Not once, in any run, ever.

Defect two: the && was decorative.

Look at the bash column. Bash split correctly, found the real defect, and printed Safe to publish. anyway.

done && echo "Safe to publish." binds to the exit status of the for loop, and a for loop’s exit status is the status of the last command in its body. The last command in the body was echo "PROBLEM: ...", which succeeded. Zero. So the “Safe” message printed because the failure message printed successfully.

Turn the grep sense around and it gets worse: the loop’s status becomes whatever the last iteration’s grep returned, so whether “Safe” appears is decided by the content of whichever file happened to be last. The message was never connected to the result.

Nothing was actually broken. I checked the whole corpus afterward with a correct tool and every link resolved. That was luck. It was not verification, and the distinction is the entire point of this piece. For weeks I had a green light wired to a switch that was not connected to anything, and I was steering by it.

The variant that is worse: a check that prints the passing result

The loop above printed a reassuring message. There is a common variant that does something worse, and it is the default shape of every quick verification anyone writes in a shell. Suppose you are sweeping a couple of files for strings that must never appear on the site:

P="fileA.md fileB.md"
grep -n -E '<email regex>' $P || echo "  none"
grep -n -E 'FORBIDDEN' $P || echo "  none"

Same cause: $P is not word-split, so it becomes one filename that does not exist. But look at what this version prints, which is worse than the first case in a specific and instructive way:

  none

The check did not merely fail to run. It printed the passing result.

grep could not open the file, so it exited non-zero. || fired. And the message on the right-hand side of that || was none — the word this script prints when it has scanned the files and found no violations. A reader scanning the output for problems sees a pass. There is no distinguishing it from a real pass, because the two conditions produce byte-identical output.

Note also what does not save you here. grep writes a diagnostic to stderr, so in an interactive terminal there is a warning line to notice. In the place you would actually deploy a check like this — a git hook, a cron job, a launchd agent, CI — stderr is redirected or discarded and nobody reads the transcript. The only thing anyone ever sees is the word none.

That is the anti-pattern in one line:

cmd || echo "clean" conflates “ran and found nothing” with “did not run.”

It is the default shape of every quick verification anyone writes in a shell, and it is structurally incapable of distinguishing the two conditions you most need distinguished.

grep gives you the fix for free and almost nobody uses it: its exit status is three-valued, not two. Verified:

found          -> 0
not found      -> 1
cannot read    -> 2

So handle all three:

FILES=(a.md c.md missing.md)
worst=0
for f in $FILES; do
  out=$(grep -n -E 'TKTK' "$f" 2>&1); rc=$?      # unfinished-copy placeholder
  case $rc in
    0) echo "VIOLATION $f: $out"; worst=1 ;;
    1) echo "clean     $f" ;;
    *) echo "ERROR     $f: $out"; worst=2 ;;
  esac
done
exit $worst

Run against one clean file, one containing a violation, and one that does not exist:

clean     a.md
VIOLATION c.md: 1:Rates start at TKTK per hour.
ERROR     missing.md: grep: missing.md: No such file or directory
script exit=2

Three outcomes, three messages, three exit codes. ERROR does not look like clean, and the script’s own status says “I could not do my job” rather than “your files are fine.”

There is a general lesson underneath the specific one. Knowing about a footgun is not the same as having a check that catches it. Nobody writes cmd || echo "clean" because they believe an unreadable file is the same as a clean file; they write it because it is the shortest thing that produces output, and the output looks like success. That is the argument for the discipline further down, and for getting verification logic out of the shell entirely.

The shell fixes, and why you should not stop there

For completeness, all four of these produce three iterations under zsh. I ran each:

for s in ${=CLEARED}; do ... done      # explicit split operator
CL=(alpha beta gamma); for s in $CL    # an array, which is the idiomatic answer
setopt shwordsplit                     # make zsh behave like bash, globally
print -r -- $CLEARED | IFS=' ' read -rA arr

The array is the right answer within the shell. setopt shwordsplit is the tempting one and I would avoid it: it changes behavior for everything in the script, including code you paste in later that was written expecting zsh semantics.

But the real lesson is one level up. I did not know zsh behaved differently. I have written shell for twenty years and I would have told you confidently that unquoted expansion splits. The failure was not that I did not know; it was that I bet a verification step on knowing.

Here is the rule I now use, and I think it is the correct rule for any lawyer building automation:

Shell is fine for orchestration — run this, then that, in this order. Shell is a bad place for anything whose result you will rely on. If the output of a step is going to be treated as a fact, get out of the shell.

Not because shell is bad, but because its semantics vary by interpreter, by version, by option flags, by whether a variable happened to contain a space, and by whether the login shell changed under you in a macOS upgrade. Every one of those variations is silent. None of them raise.

Exit status is the contract

The rewrite is thirty lines of Python. The important part is not the language. It is that it exits non-zero.

#!/usr/bin/env python3
"""linkcheck.py — no article may link to an article that is not published.

Exits 0 only if it checked something and found nothing wrong.
Exits 1 on a defect. Exits 2 if it could not do its job.
"""
import re, sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
COLLECTIONS = {"practice-lab": ROOT / "src/content/practice-lab",
               "news": ROOT / "src/content/news"}

INTERNAL = re.compile(r"\]\(/(practice-lab|news)/([a-z0-9-]+)/?\)")
DRAFT = re.compile(r"^draft:\s*(true|false)\s*$", re.M)

def is_published(collection, slug):
    """True / False, or None if the target does not exist at all."""
    path = COLLECTIONS[collection] / f"{slug}.md"
    if not path.is_file():
        return None
    m = DRAFT.search(path.read_text(encoding="utf-8"))
    return m is not None and m.group(1) == "false"

def main(argv):
    for name, d in COLLECTIONS.items():
        if not d.is_dir():
            print(f"FATAL: {d} does not exist", file=sys.stderr)
            return 2

    if argv:
        targets = []
        for slug in argv:
            hits = [d / f"{slug}.md" for d in COLLECTIONS.values()
                    if (d / f"{slug}.md").is_file()]
            if not hits:
                print(f"FATAL: no article named {slug!r}", file=sys.stderr)
                return 2
            targets.extend(hits)
    else:
        targets = sorted(p for d in COLLECTIONS.values() for p in d.glob("*.md"))

    if not targets:
        print("FATAL: nothing to check", file=sys.stderr)
        return 2

    links = defects = 0
    for path in targets:
        for collection, slug in INTERNAL.findall(path.read_text(encoding="utf-8")):
            links += 1
            state = is_published(collection, slug)
            if state is None:
                defects += 1
                print(f"BROKEN  {path.name} -> /{collection}/{slug}/ (no such article)")
            elif state is False:
                defects += 1
                print(f"DRAFT   {path.name} -> /{collection}/{slug}/ (target is draft: true)")

    print(f"checked {len(targets)} file(s), {links} internal link(s), {defects} defect(s)")

    if links == 0:
        print("FATAL: examined 0 internal links - the check proved nothing",
              file=sys.stderr)
        return 2
    return 1 if defects else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Run against the live content directory:

checked 192 file(s), 432 internal link(s), 0 defect(s)
exit=0

Four design decisions in there, and they are the transferable part.

1. Three exit codes, not two. 0 = checked and clean. 1 = checked and found a defect. 2 = could not perform the check. That third state is the one people leave out, and it is the one that caused the original bug. “I could not do my job” must never be reported as “your work is fine.”

2. A missing input is fatal, not skipped. Pass a slug that does not exist and you get exit 2. The seductive alternative — skip it, keep going, report on what you found — is precisely how a typo in an argument turns into a clean bill of health.

3. It refuses to succeed having proved nothing. If it examined zero internal links, exit 2 with an explanation. That guard is four lines and it is the single most valuable thing in the file.

4. It reports its own denominator. 192 file(s), 432 internal link(s). Those numbers are how a human notices that today the check looked at eleven links instead of four hundred. A check that only ever prints a verdict gives you nothing to be surprised by.

Because it exits properly, it composes:

# .git/hooks/pre-push  (chmod +x)
#!/bin/bash
set -euo pipefail
python3 linkcheck.py

# .github/workflows/check.yml
- run: python3 linkcheck.py

Neither of those needs to know anything about link checking. They need one thing: a non-zero exit. A check that prints a message and always exits 0 cannot be automated at all — it can only be read by a person who is paying attention, which is the resource you were trying to conserve.

Test the test by breaking it on purpose

This is the discipline the whole piece exists for.

Introduce the exact defect the check is supposed to catch, and confirm it fails. Not “confirm it passes on good input” — anything passes on good input, including a script that does nothing.

Here is the recipe, run for real:

1. Create a temporary article containing one link to a slug that does not exist and one link to an article whose frontmatter says draft: true.
2. Run the check on it.
3. Confirm both the message and the exit code.
4. Delete the temporary files.

BROKEN  zz-temp-linkcheck.md -> /practice-lab/an-article-that-does-not-exist/ (no such article)
DRAFT   zz-temp-linkcheck.md -> /practice-lab/zz-temp-draft-target/ (target is draft: true)
checked 1 file(s), 2 internal link(s), 2 defect(s)
exit=1

Then the third state, on a file with no internal links at all:

FATAL: examined 0 internal links - the check proved nothing
exit=2

Thirty seconds of work. It would have caught the original bug on day one, because the original bug had exactly one observable symptom: it was incapable of printing anything but “Safe.” A single deliberate defect would have exposed it immediately.

Two refinements that make this stick.

Keep the broken input as a fixture. Do not delete the poisoned file after you have watched it fail — check it into a fixtures/ directory and add a test that asserts the check rejects it. Now the discipline is permanent instead of a thing you remembered to do once.

Write the failing test first when you can. The reason test-first is more than a slogan here: a test written after the code, against code you believe works, is a test you have never seen fail. It is documentation of your belief, not evidence.

And there is an obligation-shaped version of this question for anything in a law office that is supposed to stop you: when did this last say no? A conflicts screen that has never returned a hit, a deadline validator that has never rejected a date, a privilege filter that has never withheld a document. Each of those may be correct. Each of them is also indistinguishable from a return True, and you have no evidence for which one you own.

The rest of the silent-success family

The shell bug is one member of a large family. All of these report success without having done anything:

The empty loop over an empty list. The most common. I ran this:

FILES = list(Path("posts").glob("*.markdown"))   # wrong extension

def check_all():
    return [f.name for f in FILES if "draft: true" in f.read_text()]

print("files considered:", len(FILES))
print("verdict:", "SAFE" if not check_all() else "STOP")

files considered: 0
verdict: SAFE

A glob typo — .markdown for .md — and the function is perfectly correct and perfectly useless. Every list comprehension over an empty list returns an empty list, and an empty list of problems reads as “no problems.” The guard:

if not FILES:
    raise SystemExit("FATAL: matched 0 files - the check proved nothing")

The grep that matches nothing. Same shape, and the second incident above is one instance of it. Another: grep -q "Rule 11" brief.pdf matches nothing because a PDF is not text, so a check for “does this brief mention Rule 11” answers “no” with total confidence, forever. Any grep whose “not found” branch is a success message needs its exit status inspected, not its output.

The try/except that swallows. except Exception: pass is the single most efficient way to convert a crash into a wrong answer. If you must catch broadly, log the exception and re-raise or return a failure state — never return the success state.

The filter that filters everything. A date-range predicate with the boundaries reversed, a status filter for a value that was renamed, an AND where an OR belonged. Zero rows out. Downstream, zero problems.

The assertion in a code path that never runs. The validation is right there in the file, reviewed, correct — behind a condition that is never true, or in a function nobody calls anymore. It looks like coverage and it is decoration.

And set -euo pipefail, which is necessary and not sufficient. Use it in every script:

• -e exit on error
• -u error on unset variables
• -o pipefail a pipeline fails if any stage fails, not just the last

That last one is worth demonstrating, because it surprises people:

'false | cat' exit=0
with pipefail    exit=1

By default a pipeline reports only its final command’s status, so generate_report | tee out.txt succeeds whenever tee succeeds — including when generate_report crashed and produced nothing.

But be honest about the limits. I re-ran the original loop under set -euo pipefail and it still ran once, still checked nothing, and still exited 0. Word splitting is not an error. || catches the failure before -e sees it. set -euo pipefail protects against errors; it does not protect against a script that is doing the wrong thing correctly.

The professional part

The value of a verification step is entirely in its willingness to say no.

That is not a metaphor. A check that returns “clean” on every input carries exactly zero information, and it is worse than having no check, because its existence is what let you stop looking. The green light did not just fail to help. It actively substituted for the attention I would otherwise have paid.

So the question to put to every automated safeguard in your practice is not “is it running.” It is “when did it last refuse something, and have I ever watched it refuse?”

If the answer is that it has never said no, that is not a track record. It is an absence of evidence, and it should buy suspicion rather than trust.

Sources

Every behavior described above was reproduced by direct execution on macOS while writing this article; the outputs shown are actual outputs, not illustrations.

• zsh 5.9 (arm64-apple-darwin25.0), the default login shell on this machine, compared against bash on identical input. zsh does not word-split unquoted parameter expansions in a for list; bash does. The same is true of an unquoted expansion supplied as a command’s argument list, which is how the second incident occurred.
• ${=VAR}, array assignment, setopt shwordsplit, and IFS=' ' read -rA each verified to produce three iterations under zsh.
• for … done && echo verified to bind to the exit status of the loop’s last body command in both shells.
• grep exit status verified three-valued on this system: 0 match found, 1 no match, 2 file could not be read. The three-way case handler above was executed against a clean fixture, a violating fixture, and a missing file, producing the three distinct outputs and exit status 2.
• cmd || echo "clean" verified to print the clean message and exit 0 when cmd failed for an operational reason rather than a substantive one.
• false | cat verified to exit 0 by default and 1 under set -o pipefail.
• set -euo pipefail verified not to prevent the original defect.
• linkcheck.py executed against this site’s live content directory (192 files, 432 internal links), against a deliberately poisoned fixture (exit 1), and against a file with no internal links (exit 2).

General commentary on engineering practice for law firm tooling. Not legal advice. All example files and slugs are invented for this article; no client information, matter, or document appears anywhere in it. Questions about anything here: Send us a message or 612-470-6529.