A release-engineering account of the rollbacks, missing production conditions, deterministic canaries, interpreter mismatch, and final process-start diagnosis that led to a healthy atomic deployment.
The validated candidate initially existed only in a dirty development tree containing unrelated work. The installer also upgraded a live virtual environment in place. Neither could establish which bytes were tested, packaged, loaded, or available for rollback. The release process moved to clean worktrees, exact-scope commits, immutable tags, wheel and installed-file hashes, release-specific environments, atomic systemd drop-ins, maintenance ingress, and executable-only rollback.
| Stage | Observed failure | Correction |
|---|---|---|
| Pre-RC | Validated bytes mixed with unrelated dirty-tree work; live venv mutated in place. | Isolated commit/tag, immutable environment, source→wheel→installation attribution. |
| RC1 | Model-selected smoke tests chose the wrong tools; a compressed stale answer passed; recovery amplified latency to hundreds of seconds. | Production-history fixture, bounded directional reuse check, deterministic authenticated canary hooks. |
| RC2.1 | Background exit was durable, but output emptiness was not reconstructably typed; maintenance producer coverage and mutation-cause prose needed proof. | empty|nonempty|unknown plus bytes, producer matrix, fail-closed maintenance, cause-compatible prose. |
| RC2.2 | The validated venv’s Python symlink resolved to the host interpreter, so copying it could not preserve the reviewed Python 3.14.6 identity. | New release validated explicitly on the production-compatible Python 3.14.4 runtime. |
| RC2.3/2.4 | Cancellation erased a failed tool outcome; deterministic failure canary still entered long model recovery. | Job-scoped action preservation through cancellation and fail-fast infrastructure assertions. |
| RC2.5 | Background canary proved its lifecycle, then unnecessary model/reviewer recovery made it appear failed and slow. | Canaries terminate as soon as their durable acceptance facts exist. |
| Final blocker | Background child could not start. | Diagnosed and removed a canary-only inaccessible working-directory override. |
0700, then supplied that path as the service child’s working directory. The unprivileged service process could not traverse it, and process creation raised PermissionError [Errno 13].The executable itself was valid. Python subprocess behavior, /usr/bin/true, journal paths, systemd restrictions, and the normal workbench path were not the cause. Ordinary background jobs inherit the service working directory. The fix removed only the canary-specific cwd override and added a regression reproducing inaccessible workspace behavior.
Deployment canaries became infrastructure assertions rather than miniature conversations. Mutation ends when the expected adapter failure and zero writes are durable. Background ends when process settlement, exit code, output state/bytes, and journal reconstruction are durable. Stance ends after authoritative lookup and a before/after mutation check. Recall ends when typed retrieval state is established. Journal provenance ends when required receipts and linkage exist. Model prose or reviewer approval is required only when prose or review is the object being tested.
exited, exit 0, empty, 0 bytes, reconstructed, valid chain.Rollback changes the executable release only. Databases are not restored unless corruption is proven, preserving valid state created around the switch.
| Tag | kairo-validated-20260822-rc2.6 |
|---|---|
| Commit | 55bb7bc2a1a93178877d0040d7513ecf9dfe218a |
| Tree | 54876853032b5df6e5ff3730bf5f9d5e061b8f0b |
| Wheel SHA-256 | 867a6dced39c94ade1fd3091d184782a0c730d15ab88d43bcfdb48bec6ad753b |
| Manifest SHA-256 | cd628c1f67eb816df20962ce28546ba4d386beea4a30408fcff7b771a02e967b |
| Database schema | 16; no migration |
| Final state | DEPLOYED_HEALTHY |
SQLite integrity was ok, stance state did not change because of deployment or smoke inspection, and normal traffic reopened successfully.
A later two-file RC2.7 successor corrected ordinary continuing-topic projection without reopening the sprint architecture. It was packaged from RC2.6, passed the directly affected tests and installed-wheel smoke, deployed through the same maintenance/snapshot/atomic-switch mechanism, and passed live service, storage, and recall checks. RC2.6 remains the release that closed the engineering sprint described here.
It does not mean every optional canary is a scientific test of model quality, that no future defect exists, or that Kairo must take any particular philosophical position. Deployment acceptance concerned loaded identity, storage integrity, core foreground operation, typed lifecycle truth, retrieval-state truth, and preserved self-authorship.