Inference benchmark · October 13–17, 2026

PXA vs llama.cpp: why PXA wins on Kairo’s V100 workloads

Topics: Inference Engineering · Case Studies  ·  Subject: Kairo

The same model weights. Faster prompt processing. Shorter measured requests. A comparison grounded in recorded results, including the configuration where PXA lost.

Historical benchmark synthesis · Prepared October 17, 2026 · “llama” here means the llama.cpp inference engine, rather than Meta’s Llama model family.

The result

PXA was superior for the measured V100 configurations. In the strongest matched 35B comparison, it processed input 1.53–1.68 times as fast as the pinned upstream llama.cpp build distributed with V100 Kit. Generation throughput was 1.04–1.15 times as high. PXA finished every tested input length sooner. [E1]

The 9B and 0.8B studies provide additional evidence of throughput gains, with different methods and qualifications. They are separate experiments, not a pooled score. The RTX 3060 Hammer trial supplies an important boundary: the tested patched PXA build lost both on measured speed and on complete tool-workflow success. [E2–E4]

35B: the strongest matched comparison

Both arms used the same post-trained Qwen 35B-A3B Q6_K model and Q8 vision projector, with full GPU offload across two Tesla V100-PCIE-32GB GPUs. The devices were connected through PHB, with no NVLink; six other model processes remained resident. PXA v3.1/build 6600 was compared with V100 Kit’s upstream llama.cpp build 9611, revision 02182fc5. This was not a test of the kit’s ik_llama backend or its other profiles. [E1]

A percentage increase in throughput differs from a percentage reduction in time: the 32K reference takes about 49% longer than PXA, while PXA takes about 33% less time than the reference. Both describe the same pair of observations.

Why the advantage matters

1. Input processing is the larger measured gain

Prefill processes the supplied prompt before generation. In uncached requests with substantial input, speeding this phase can save more time than a small improvement in output generation. The 35B data show precisely that pattern: a 53–68% prefill-throughput gain accompanies a smaller 4–15% decode gain. The longest tested request saves about seven seconds. This is an explanation of the measured timing pattern; the study did not isolate a particular kernel as its cause. [E1]

2. The improvement did not require different model weights

The 35B arms used identical model and projector bytes, and the timing comparison held input token arrays and output counts fixed. The observed gain therefore demonstrates an engine/build advantage for this configuration without changing the trained model. Different builds still produced different output hashes; identical weights and deterministic sampling settings do not prove identical text or semantic equivalence. [E1]

3. The tested configuration fit the operating constraints better

At the then-serving 2:1 split, V100 Kit crossed the required 2,048 MiB GPU1 free-memory floor during startup, reaching a sampled 1,627 MiB. It was stopped before timing. The accepted comparison used a common 5:2 split for both engines; sampled free-memory minima were 3,833 / 2,199 MiB, with no recorded OOM-counter change. This supports the historical configuration choice, not a universal claim that PXA uses less memory. There are no accepted kit timing results at 2:1. [E1]

9B and 0.8B: supporting results

The smaller-model comparisons used the installed incumbent llama.cpp binary, whose version reported 0 (unknown). Its source revision was not established. These results must not be described as comparisons against the latest upstream llama.cpp. [E2, E3]

PXA’s 9B decode gains were 19.6%, 24.2% and 28.9%; prefill gains were 7.6–12.8%. At 2K and 8K, PXA also generated fewer tokens, so the observed completion savings are not equal-output latency measurements. The test allocated 17,408 context tokens, rather than qualifying the full 131,072-token serving configuration. The stock arm also loaded two zero-scale adapters; the PXA arm did not. Broad role-specific quality was not established. [E2]

The helper’s 256- and 1,024-token inputs showed 21–24% faster decode. At 3,072 tokens, the roughly 2.02× result included 72 accepted n-gram draft tokens per sample; the continuation repeated material from the prompt. PXA’s automatic speculation contributed to that workload-dependent result. No accepted draft tokens were recorded in the shorter cases. [E3]

Short natural-EOS checks were mixed: the first arithmetic request after startup took a median 165 ms on PXA versus 72 ms on llama.cpp; subsequent literal JSON took 88 versus 145 ms. Math always preceded JSON, so task type and initialization state were confounded. These are minimal compatibility checks, not a representative helper-task latency evaluation. [E3]

Where PXA lost: Hammer on RTX 3060

Hammer decode is normalized on both engines as (predicted_n − 1) / predicted_seconds, because their native rate-reporting conventions differ. These fixed-length performance probes are separate from workflow-success checks.

This used upstream revision 797de63 and a public PXA 33d292ae base rebuilt for sm86 with sampler-isolation and reserve-guard patches. It was a different binary/library stack from the V100 package. The planned A/B/B/A comparison stopped after the first complete A and B blocks under the malformed-call stop rule. Its direction is adverse evidence, but counterbalancing was not completed and the exact cause of malformed JSON remains unresolved. The original conditional workflow was not completed by either backend. [E4]

Methods, controls and limits

35B matched settings

131,072 allocated context; one slot; 10 generation and batch threads; batch 2,048 / microbatch 512; Q8 K/V; explicit flash attention; layer splitting at 5:2; CPU vision projector; speculation disabled. Temperature 0 and seed 42. The kit source was pinned at 0ff9df3a and its Linux resident-Q6 upstream engine at 02182fc5. PXA reported build 6600 / b0fb173d3f; equivalence with the public source tag is not asserted. [E1]

Fresh processes ran PXA → kit → kit → PXA. Each passed natural-EOS arithmetic and literal-JSON checks before warmup. Identical uncached token arrays produced four measured 128-output-token samples per engine and input length. End-of-generation suppression was confined to disposable fixed-length timing requests, with all five model EOG IDs constrained identically. Process startup was excluded. Other engine internals/defaults remained build-specific. [E1]

All eight minimal 35B checks passed. End-to-start PXA prefill drift was at most approximately 1.1%, and decode drift approximately 2–3%; the small 1K decode gap deserves caution. The largest input tested was 32,768 tokens, despite the larger allocated context. This is neither a full-window benchmark nor a vision-quality evaluation. [E1]

Retained failures and production qualifications

The first 35B attempt stopped at 80 output tokens and supplied no accepted timing rows; its recovery controller required a manual restart after mishandling stopped-service status. The 2:1 memory-floor failure was a separate attempt and is excluded from the matched 5:2 medians. The helper’s initial same-port retry failed to bind and was excluded. Later successful matrices used the corrected controls. [E1, E3]

Separate October 13 incident records initially associated repetition failures with cancellation. Subsequent causal testing identified request-local EOS sampler bias leaking across slot reuse: ordinary cancellation did not reproduce it, while a completed forced-EOS-suppression request did. A request-isolation gateway was subsequently qualified; the reserve-guard issue also required separate work. These benchmark results do not certify an unqualified PXA package for production. This report synthesizes historical experiments and makes no claim about today’s running PIDs or configuration. [E5]

What would strengthen the conclusion

A larger counterbalanced comparison on representative user requests, measured full-agent latency, long-context quality tests, sustained concurrent-load trials, locked or recorded clock behavior, and distribution-wide tool/JSON checks. The existing shared-hardware samples do not establish confidence intervals or isolate the kernel responsible for each gain.

The case for PXA

Choose PXA for the demonstrated V100 advantage, and qualify each deployment. The 35B comparison shows a substantial, repeatable-in-this-matrix improvement in uncached prompt processing and request time with the same model weights. Smaller-model results reinforce the throughput case. The Hammer failure defines a boundary rather than disappearing from the record.

The evidence supports PXA as the better-performing engine for Kairo’s tested V100 configurations. It does not support assuming that every PXA build will outperform every llama.cpp build on every model or GPU.

Evidence and independent checking

The downloadable evidence contains all 66 selected V100 timing samples, recomputed medians, original-file hashes, and the Hammer block aggregates. It excludes raw prompts, responses, host addresses, credentials, process inventories and service configuration. Hammer aggregates are labelled as such; they are not individual request samples. A file hash identifies the retained original but does not make that original publicly downloadable.

Evidence SHA-256: f739aa41d68e9d12574076b5085f5abc6be71b2885a7a1976582caf77a0afb6b. Source-data commit: dfb801784d91325293506e52ea73e22aaf06d4c8.

E5: artifacts/kairo-pxa-loop-fix-20261013/REPORT.txt, artifacts/kairo-pxa-sampler-fix-20261013/REPORT.txt and artifacts/kairo-pxa-reserve-guard-20261017/REPORT.md. Historical operations records supply the reliability qualifications, not additional speedup samples.