Inference-engine forensics · October 8–20, 2026

Forced Full Prefill: Why Kairo's Hybrid 35B Re-prefills Every Turn

Topics: Inference Engineering · Case Studies  ·  Subject: Kairo

A complete forensic record of the “forcing full prompt re-processing” warning in Kairo's production PXA runtime: what the logs measure, what the pinned build actually does, what a controlled CPU-only experiment proved, and why no production change was made.

MEASURED marks numbers taken from production logs, API timings, or the isolated experiment. MECHANISM marks behavior read from the pinned build's own source. EXPERIMENT marks results of the shadow test matrix. DECISION marks the operational conclusion. UNRESOLVED marks an evidential limit.

Executive summary

Kairo's primary chat model, a hybrid attention/recurrent 35B MoE served by a pinned llama.cpp-derived server (PXA v3.1-guardfix), re-processed essentially its entire prompt on almost every conversation turn. Over a five-day production log window, 3,343 turns logged forced full re-processing against only 128 checkpoint restores. The warning blames SWA eviction; in this build that label is wrong.

The verified root cause is a triple: (1) the model's recurrent layers give each sequence a single state position that cannot be rewound, so prefix reuse legally requires a context checkpoint taken strictly before the divergence point; (2) the build's default checkpoint policy creates snapshots only near the end of each prompt; and (3) one shared inference slot interleaves heterogeneous traffic, and every low-overlap interloper erases the checkpoints the next conversation turn would need. A controlled experiment on the same binary confirmed each part independently — and confirmed that the obvious configuration fix (PXA_CKPT_PROMPT_POLICY=0) is neutralized by the erase cascade under interleaved traffic.

Scope and method

• All production work was read-only: log parsing, hash verification, and metric counters. No restarts, configuration, cache, service, model, or route changes were made at any point.
• The build under test was identified by hash: llama-server sha256 fe68c417…9bdc, libllama.so edcb3af2…0d58, release manifest source = PXA fork (an ik_llama.cpp descendant) at commit 33d292ae plus the tracked pxa-reserve-guard-amortized v2 patch — which touches only graph-memory reservation, never cache logic. The server's own checkpoint source was read at the pinned commit.
• Production evidence came from the server's WRN-level checkpoint log lines — every lifecycle event (create, erase, restore, force) is logged with exact positions, so no payload capture was needed and no prompt text left the machine.
• The follow-up experiment ran the identical pinned binary CPU-only with a same-architecture 0.8B hybrid model, fully isolated (no GPU device visible, dedicated cores and ports, dedicated directory, later deleted). No production process or file was touched.

Production measurements

All lifecycle events over the ~4.5-day span of the two retained 35B error logs (Oct 15–20, 2026), with the 9B gateway on the same build as a cross-check:

Across 3,292 partial-reuse turns on the 35B, the measured token-level longest common prefix between each request and the slot's cached prompt was bimodal: percentiles (min, p10, p25, median, p75, p90, max) of 1, 3, 3, 32, 3,734, 9,923, 155,259 against prompt sizes median 6,552, max 258,037. 57% of turns shared fewer than 512 tokens; 24% shared 4,096+. Sum of reusable prefix tokens: 12.39M of 66.23M prompt tokens (18.7%). An earlier engine-side capture agreed: an 8 Oct conversation turn diverged at token 23,041 of 33,443.

Corroborating measures: 5.74M prompt tokens costing 3,325 s of prefill in an 18.4 h metric window at ~2,100 tok/s; KV-cache usage ~2.4%; 100 durable turns on 16 Oct all reported cached_prompt_tokens = 0; each created checkpoint cost 62.9 MiB and ~63 ms, while each restore cost only 12–37 ms.

The mechanism in the pinned build

MECHANISM Four code facts, read from the pinned server source and verified against the binary's strings and production log lines:

1. Hybrid rollback is mandatory. The model family mixes 10 attention layers with 30 gated-delta-net recurrent layers. A recurrent sequence has one state position; trimming KV cells cannot move it back. The fork's rollback rule therefore permits prefix reuse only by restoring a checkpoint whose snapshot position is strictly below the re-entry point; otherwise the slot resets to zero and logs forcing full prompt re-processing due to lack of cache data (likely due to SWA…). For this build the SWA attribution in the message is a mislabel: cell eviction is not the trigger; absent checkpoints are.
2. Checkpoints sit only at prompt ends. PXA_CKPT_PROMPT_POLICY (ON by default) skips mid-prompt checkpoints; only two to four snapshots within ~517 tokens of the prompt end are created. PXA_CKPT_PROMPT_END (OFF by default, unset in production) would add one at the prompt end. Production flags --ctx-checkpoints 64 (capacity; never binding) and -cram 0 (prompt-shift cache, unsafe for this arch by the fork's own bug note) were confirmed non-causal.
3. The erase cascade. Every partial-match turn deletes all checkpoints above its re-entry position. A turn whose prompt shares nothing with the previous one (measured: p25 = 3 tokens, often the BOS alone) deletes the entire ring — and a zero-overlap interloper performs the cascade without ever logging a forcing line, so the erasures outnumber every visible symptom.
4. Traffic shape. One shared slot interleaves Discord conversation, memory-worker jobs, research jobs and vision turns. Kairo's own prompts are highly stable within a stream (prefixes up to 155k tokens measured), but re-rendered inline blocks push the conversation divergence point deep into the prompt — above the tail-only checkpoints and below everything the cascade left behind.

Upstream review: issue #20225 (predicate-era checkpoint bug) was fixed differently in this fork; #21831 closed without a fix; #22746 independently documents the identical interleave-erases-ring signature on a Qwen3.6 agent harness, and its candidate fix was never merged and is absent from Kairo's build.

Verified root cause

MEASURED The forced-full-prefill steady state requires all three conditions, and all three hold in production:

• Placement failure: with tail-only checkpoint creation, no snapshot exists below a mid-prompt divergence. The shadow experiment reproduced this with production defaults at 56% divergence: 100% of checkpoints sat above the re-entry point; the turn fully re-prefilled.
• Retention failure: even with mid-prompt checkpoints enabled and proven to restore, an unrelated interloper between the two turns erases them before the next turn arrives. In the experiment, enabling every available lever changed nothing once an interloper was inserted.
• Baseline loss: 57% of production turns share under 512 tokens with the previous request, so they are effectively cold starts and continuously destroy the ring for everyone else.

The warning count (3,343) understates the cost because zero-overlap turns re-prefill silently. The real cost is visible in the metric counters: nearly every token of every prompt is evaluated on every turn.

Controlled shadow experiment

EXPERIMENT The pinned binary, CPU-only, zero GPU (verified: CUDA initialization reports no device), same qwen35 hybrid architecture at 0.8B, production cache-mechanic flags (-b 2048 -ub 512 --kv-unified --ctx-checkpoints 64 -cram 0 -np 1), greedy decoding, fresh process per cell. Synthetic prompts (no Kairo content): A = 4,560 tokens; C = same first 2,531 tokens then diverges (~56%); B = unrelated ~99-token prompt with zero overlap. Sequences: A→C and A→B→C, each under four configurations:

What the matrix established:

• Placement fix works when tested alone. Disabling the prompt policy created mid-prompt checkpoints; restore re-entered exactly at checkpoint position +1 and saved 2,048 tokens — 41.4% of the turn's prefill — for a 10 ms restore.
• Granularity is the decode batch, not the configured interval. Mid-checkpoints landed at -b 2048 batch boundaries (positions 2047, 4095), not every 512 tokens. A first-round prompt with a 1,956-token divergence missed its only checkpoint by 91 tokens and still forced. At production 45k prompts this means ~22 checkpoints per turn with a worst-case 2,048-token residual.
• Retention failure dominates. With the interloper inserted, every configuration — including both levers enabled — re-prefilled C in full. The cascade from a zero-overlap request deleted A's mid-prompt checkpoints before C arrived. No available configuration prevents this.
• The prompt-end lever is inert for this failure mode. It armed and created its extra checkpoint, but a prompt-end snapshot can never sit below a mid-prompt divergence. Append-style continuation already restores under production defaults (a probe turn with a 6,341-token shared prefix re-processed only 206 tokens — 96.7% saved).
• No incorrect state reuse, anywhere. Greedy completions were byte-identical across every configuration: the restored turns matched the full-recompute turns exactly. The interloper's own completions were never poisoned, and in interleave cells the engine correctly refused every remaining checkpoint. The hybrid restore path is sound at this scale; only its placement and retention are wrong for this traffic.

Avoidable-latency bounds

MEASURED Upper bound from production logs: 12.39M of 66.23M prefill tokens (18.7%) lay inside reusable prefixes over five days; at the observed 2,100–2,300 tok/s that is roughly 5,200–5,700 s of avoidable prefill, concentrated in the 24% of turns with 4k+ token overlap. The single-turn experiment realizes 41% prefill saving on a consecutive same-stream pair at 56% stability. Under current interleaved traffic, however, a restore can recover at most the interloper's own prompt length (typically ~6.5k tokens), so naive configuration changes recover near zero of the bound — which is exactly what the experiment measured.

Unrelated TTFT components (gateway, queue, network: ~4–14 s per turn on 16 Oct durable data) and one anomaly (~44.8 s prefills of ~1.2k-token turns, candidate: CPU vision projection on multimodal turns) are not addressable by checkpoint work at all.

Unresolved items

UNRESOLVED Three limits are stated rather than papered over: (1) the exact first-diverging component of Kairo's serialized prompt (tool-schema churn versus identity headers versus inline runtime snapshots) could not be isolated without payload capture, which the authorized observation surface did not include; log-derived LCP and the 8 Oct engine-side capture bound the effect but not its authorship. (2) The pathological ~44.8 s small-prompt prefills were not fully attributed. (3) The 2,048-token checkpoint granularity floor and its interaction with a smaller batch size were not qualified.

Decision and revisit conditions

DECISION Production remains unchanged. On this evidence:

• PXA_CKPT_PROMPT_POLICY=0 alone is not worthwhile: measured benefit under interleaved traffic is ~0, while it adds ~20 checkpoint creations per production turn (~1.3 s of device drain and ~1.3 GiB of PCIe traffic per turn at 35B scale).
• PXA_CKPT_PROMPT_END=1 alone: skip — inert for Kairo's mid-prompt divergence pattern.
• The measured ceiling (~18.7% of prefill; 40%+ per consecutive conversation turn) becomes reachable only if conversation turns are served consecutively on a slot — i.e. client-to-slot affinity or workload routing on the shared lane. That is an admission-control/routing change requiring explicit authorization and full re-qualification; if ever authorized, PXA_CKPT_PROMPT_POLICY=0 is the pre-validated companion lever.
• Engine-side alternatives (moving re-rendered volatile blocks after the stable transcript so existing tail checkpoints become usable) touch context-construction semantics and remain separately gated.
• Any production adoption changes the pinned build's runtime configuration and requires re-running the qualification harness, including the erase-cascade replay used here.

Provenance and preservation

Every number above is reproducible from artifacts committed to the Kairo main branch:

• Measurement chain: artifacts/kairo-cache-phase3-20261008/ (8 Oct prefix captures), artifacts/kairo-engine-preflight-20261016/ (durable turn timings), artifacts/pxa-record-reconciliation-20261020/ (runtime identity verification).
• Diagnosis: artifacts/pxa-forced-reprocess-20261020/REPORT.md (commit 0b8f76b9) — pinned-source reading, log-parse methodology, and full counts.
• Experiment: artifacts/pxa-shadow-ckpt-exp-20261020/ (commit 2f153f5f) — this report's HTML source directory also holds per-cell meta.json results for all 14 cells plus both runner scripts; full server logs are preserved off-repo at /tmp/opencode/pxa_shadow_exp/logs/ on the engineering workstation.
• Patch provenance: artifacts/kairo-pxa-reserve-guard-20261017/ (the guardfix patch and its qualification record).

No production process, service unit, nginx configuration, model, cache or route was modified during any part of this investigation. The shadow experiment directory on the inference host was deleted after artifact retrieval; nothing remains there.