Measuring a regression in this repo

Moved out of CLAUDE.md on 2026-08-22 so it loads when a regression is actually being measured. Nothing was cut — the text below is that block verbatim.

Known unrelated debt — and the figure is not the check. Measured on a pristine checkout of HEAD (2026-07-24, late): 81 failed / ~2980 passed / 21 errors, and the failures are not confined to test_web_api_write.py + test_web_integration.py. Pre-existing and outside the current track.

Do not quote this number as a baseline. It has now been stale twice in one file: "17 failed" understated it by ~77, and "94 / 2631 / 21" — written earlier the same day — was off by 352 passing tests within hours. The passing count also moves a few tests between runs. A debt figure is a measurement with a timestamp, and a stale one waves a real regression through as "expected".

The check that works is a set diff against a baseline you actually ran. Never a stash inside a killable command — a timeout between the stash and the pop leaves a clean tree and the whole session's work in stash@{0}, which looks exactly like a command that never started:

git worktree add -q --detach /tmp/base HEAD
python -m pytest tests/unit -q -p no:randomly 2>&1 | grep -E "^(FAILED|ERROR) " | sed 's/ - .*//' | sort > /tmp/now.txt
# THREE import roots, and a session-end assertion that nothing leaked. Both matter — see below.
cat > /tmp/leakcheck.py <<'EOF'
import os, sys
def pytest_sessionfinish(session, exitstatus):
    base = os.environ["BASELINE_ROOT"]
    leaks = sorted({m.__name__ for m in list(sys.modules.values())
                    if getattr(m, "__file__", None) and "/set-core/" in str(m.__file__)
                    and not str(m.__file__).startswith(base)})
    if leaks:
        print(f"BASELINE LEAK ({len(leaks)}): " + ", ".join(leaks[:25]), file=sys.stderr)
        session.exitstatus = 99
EOF
(cd /tmp/base && BASELINE_ROOT=/tmp/base/ \
   PYTHONPATH=/tmp/base/lib:/tmp/base/modules/web:/tmp/base:/tmp \
   python -m pytest tests/unit -q -p no:randomly -p leakcheck 2>&1 \
    | grep -E "^(FAILED|ERROR) " | sed 's/ - .*//' | sort) > /tmp/base.txt
diff /tmp/base.txt /tmp/now.txt   # empty = no regression, whatever the counts say
git worktree remove /tmp/base --force

The PYTHONPATH line and the assertion are not decoration — without them this check does not compare two versions. Measured 2026-07-24: set-core is installed editable, so its __editable___set_core_0_3_0_finder resolves set_orch to /home/…/set-core/lib from anywhere. A worktree at /tmp/base therefore ran the BASELINE TESTS against the WORKING TREE's library — a hybrid, not a baseline.

Its fail direction is what makes it expensive: the usual change is additive, so old tests still pass against new code and the failure sets come out identical. The check then reports "no regression" having compared one version with itself, and it does so most convincingly exactly when it is least earned. It only became visible when two baseline tests failed that could not fail at HEAD — the hybrid's own tell, and it appeared by luck.

So: point PYTHONPATH at the worktree's source roots, and assert where the imports came from before believing the run. This is the proxy-instead-of-the-thing class applied to a version: cd-ing into a worktree is a proxy for running its code.

And the first repair of it was itself incomplete, which is the more useful half. It set PYTHONPATH=/tmp/base/lib and asserted set_orch — one package, named by hand. Measured afterwards, prompted by an integration peer generalising the finding on their own side: this repo puts first-party code under three roots, and a raw .pth entry hard-codes modules/web to the development tree. set_project_web is imported by 10+ unit test files and was still coming from the working tree, so the "corrected" baseline was still partly hybrid. The named list was a second copy, and it drifted at the moment it was written.

Hence the session-end check above, which asserts the thing — no module loaded from any set-core checkout other than this one — instead of a list of paths somebody has to maintain. Measured on HEAD with full isolation: 0 leaks, 106 failure entries, identical to the partially-isolated run, so the earlier conclusion survives while the evidence for it is now real.

One thing this does NOT cover, raised by the same peer with their own measurement: a generated artefact can come from the other tree even when every source path is right, because it is a product, not a source (their case: a generated database client resolved from the main tree's node_modules, so worktree source ran against main-tree schema — the same hybrid, and additive changes keep it green). Measured here: set-core's Python has no generated layer (find lib modules set_tools -name '*_pb2.py' -o -name '*_generated*.py' → empty), so tests/unit is not exposed. The dashboard under web/ does have a build product, and that path has not been measured — do not assume it is clean.