Kairo Lablab.kairo.icuRestricted2026-09-21

LFM2 v2 live cycle: activation, observed use, and emergency rollback

LFM2 v2 temporarily replaced the shared former-9B supporting endpoint while the 35B remained the foreground final-response model. After an observed production-quality regression, the endpoint was restored to the known Qwen 9B configuration. This report records what was changed, what was actually observed, what was not established, and the resulting operational boundary.

Outcome

Live LFM2 interval2026-09-20 → 2026-09-21shared former-9B endpoint
Foreground final voice35B unchangednot an LFM2 final-answer deployment
Rollback resultQwen 9B restorednovexai-brain compatibility endpoint
Current LFM2 useNon-live work onlyno production claim-atomization calls

Contents

1. Scope and model roles

The live change replaced the service behind the existing supporting-model compatibility name novexai-brain. It did not replace Kairo's 35B foreground reasoning/final-response endpoint. During the LFM2 interval, the shared supporting endpoint was used by configured callers such as the controller, reviewer, coder/read assistance, router fallback, background brain/identity work, memory extraction, and Discord follow-up classification.

The separately configured linguistic route remained disabled. That distinction matters: a bad delivered Discord message can have been influenced by supporting control work, but the final-text provenance must be checked before calling it an LFM2 completion. The observed problematic delivered replies in the incident window were recorded as 35B reason-route completions, not as LFM2 foreground completions.

State boundary. The model-server cutover and rollback did not reset Lake, StateFrame, Purpose, memory, accepted jobs, or conversation history. Restoring the inference service is not a rewind of Kairo's durable state.

2. Bringing LFM2 up

At operator direction, the existing Supervisor program named kairo-9b was repointed from its Qwen 9B workload to the canonical LFM2 v2 supporting-model workload. The legacy program and compatibility alias names were retained so existing callers did not need a routing rewrite. The service ran on the existing GPU loopback endpoint 127.0.0.1:8088, exposed through the unchanged core tunnel at 127.0.0.1:18088.

ComponentActivated identity
Canonical served modelkairo-lfm2-v2-ckpt248-f07dc5c85145-q6-e560e3ccbbe5
Base revisiona3bbacd91a678b97712f0e323e52f8c24ba29542
Q6_K base SHA-256e560e3ccbbe5595b61be1d46b672206db0eb3639e515de6c5bebcd5b1a007dbc
Served F32 adapter SHA-256a62184c4a7fb0ac8b11ddfac84ef4e06c0a2cd8cc451dcce327f12a3b914782c
Context / placement128,000 tokens; later --gpu-layers 99 --n-cpu-moe 24
Compatibility aliasnovexai-brain

Startup identity, adapter scale, endpoint health, and a compatibility-alias completion were checked. A subsequent bounded placement test improved matched synthetic serving throughput, but it was explicitly a serving-placement result—not a complete quality validation for every role formerly served by 9B.

3. What the live system used it for

LFM2 was a supporting service, not the final conversational voice. That distinction was verified from the effective live configuration and individual delivery receipts. In the incident period, the actual agent process had primary role 35b, while the LFM2-backed compatibility endpoint supplied controller, reviewer, and coder support roles. The dedicated linguistic role was disabled.

Important limitation

Endpoint availability is not role-quality certification

Health checks and a short smoke demonstrate that an endpoint can serve a request. They do not establish correct controller routing, reviewer behavior, repair behavior, or the effects of those roles on a 35B final answer. The live observation window found at least one controller output-contract failure before the later conversation regression.

4. Production observation and attribution limits

During live use, Kairo produced unrelated identity/runtime-status boilerplate in response to ordinary conversational messages. This was a serious quality regression and triggered the emergency rollback request. Receipts for the affected deliveries identify the final candidate as a 35B reason-route response; one passed ordinary response-quality checks and another was delivered after a retry/repair path.

Two correlated control-path facts were also observed: a controller fast-route unavailability/fallback occurred around one bad turn, and a newly generated self-model context block of roughly 5.9 kB was inserted into each affected foreground prompt. Those facts make a control/context/retry-path regression a stronger immediate hypothesis than changed 35B weights. They do not prove that LFM2, the self-model block, or any one subsystem caused every bad reply.

Decision under uncertainty

Rollback was warranted without a complete root-cause proof

The user-visible behavior was outside acceptable operation, the known-good Qwen 9B configuration was recoverable, and leaving the changed shared support path live while attribution remained unresolved was not justified. The rollback was a containment action, not a claim that model weights alone explained the incident.

5. Bringing Qwen 9B back

The documented GPU-side rollback was run:

python3 /workspace/lfm2-9b-replacement-20260920/cutover.py rollback

It stopped only the kairo-9b Supervisor program, restored its preserved Qwen configuration, reread the program, and restarted the supporting endpoint. The restored configuration SHA-256 was:

fb7a498a1ddc0d5fc3123f46a18069d08d67ad984c9642a85b7115d92ad747cb

No application source, 35B service, memory system, state store, affect logic, or delivery behavior was changed as part of that rollback.

6. Post-rollback verification

This verifies the intended rollback target: the shared former-9B service is again Qwen 9B. It does not, by itself, prove that every outstanding foreground-quality concern has disappeared; the observed conversation/control-path issue still needs separate diagnosis before any future live model change.

7. Operational lessons and next boundary

  1. Keep final-answer provenance separate from supporting-role provenance. A 35B-delivered reply is not automatically evidence about the supporting model, and a supporting model may still influence routing or repair.
  2. Do not treat an endpoint smoke or throughput result as end-to-end behavioral acceptance for controller, reviewer, or repair roles.
  3. Before any future live substitution, require role-by-role tests, explicit controller-contract checks, and a foreground regression protocol that can distinguish final generation, prompt/context insertion, routing, and retry behavior.
  4. Keep non-live model research isolated from Kairo's production cognition and shared serving endpoint. In particular, claim-atomization experimentation must not send historical Kairo responses to the production LFM2 endpoint.
Current boundary: LFM2 work may continue only as controlled, non-live evaluation. This report does not authorize another LFM2 rollout, evidence retrieval, MiniCheck integration, production wiring, or a change to the deterministic claim-extraction baseline.

8. Evidence and limitations

The activation configuration, hashes, placement measurements, and rollback procedure are preserved in the LFM2 replacement and placement artifacts. The incident observations were established from read-only live process configuration, delivery receipts, route/candidate events, and self-model exposure receipts. Source documents retain their original timestamps and describe the historical LFM2-live state; this report adds the later rollback outcome rather than rewriting that history.

Primary records: artifacts/lfm2-9b-replacement-20260920/DEPLOYMENT.md · artifacts/lfm2-9b-replacement-20260920/THROUGHPUT.md · artifacts/lfm2-placement-20260920/REPORT.md · docs/operations/lfm2-model-inventory-2026-09-21.md

The report intentionally does not claim a final causal diagnosis. The operational facts support the rollback and identify the next investigation boundary; they do not prove that a particular model, prompt block, or repair stage was the sole cause of the observed bad replies.