Rust / original weights / measured inference
Same model.
Different engines.
Topics: Inference Engineering · Subject: Rust
Rust engine head-to-head: a transparent comparison of prefill, decode, and complete request time on the original Qwen3-30B-A3B Q4_K_M model.
November 3 tuning: parallel activation preparation
Rust now uses expXa0982fd3 with parallel activation preparation; Kairo retains its qualified a078474 native runtime. The previous port parallelized the CPU expert multiply, but activation preparation still used only1worker for up/gate and8 for down. The new Rust-only path distributes complete rows across20existing batch threads, with byte-identical numerical results. Preparation0/1 below uses the same candidate binary, isolating this change.
| Engine / mode | Input tokens | Prefill t/s | Decode t/s | Request seconds |
|---|---|---|---|---|
| V100 · expXa0982 · parallel preparation | 512 | 203.97 (199.39–207.86) | 37.28 (36.02–38.70) | 6.00 (5.77–6.07) |
| V100 · expXa0982 · parallel preparation | 4096 | 201.53 (175.58–202.04) | 32.96 (32.49–34.05) | 24.27 (24.16–27.10) |
| V100 · expXa0982 · old-preparation ablation | 512 | 190.34 (189.57–197.83) | 38.16 (37.50–38.93) | 6.00 (5.99–6.05) |
| V100 · expXa0982 · old-preparation ablation | 4096 | 189.38 (188.36–191.26) | 30.93 (29.08–32.87) | 25.83 (25.53–25.89) |
| V100 · PXA4 · tuning control | 512 | 180.82 (152.69–199.26) | 31.23 (31.07–38.33) | 6.95 (5.91–7.45) |
| V100 · PXA4 · tuning control | 4096 | 202.00 (182.11–203.13) | 30.41 (30.10–34.31) | 24.38 (24.01–26.76) |
| V100 · expXa a078 · before/after reference | 512 | 187.43 (183.49–191.37) | 37.75 (37.01–38.49) | 6.13 (6.00–6.25) |
| V100 · expXa a078 · before/after reference | 4096 | 179.34 (175.36–183.32) | 32.46 (31.40–33.53) | 26.81 (26.17–27.44) |
Kairo PID/libraries/profiles stayed unchanged; all28 captured before/after responses and token/MTP counts matched, including the known raw-EOS failure. The first post-update code timing median was20.9% slower and is retained. Three focused follow-up repetitions returned steady decode medians within−0.59% to+1.98% of baseline (code−0.14%), with identical outputs: no persistent slowdown observed in these checks. The transient delay is not attributed to a proven cause, and this is not broad quality/performance assurance.
Latin primary-engine rotation, contemporaneous incumbent brackets and identical original model/offload/thread/context limits. Sampled production-idle/resource guards retain failed/interrupted attempts. Earlier76 records remain available below and in the complete98-record downloads. Shared-host timing noise and the small fixture set limit generalization.
November 3: unified expXa vs PXA4
The earlier unification trial served both models on expXa a078474; current tuning status appears above. The Rust CPU queue/Q4_K/Q6_K port is explicitly enabled for Rust and disabled for Kairo; Kairo retains its attention and MTP profiles. Both engines below were tested as additional private clones after the switch, with the same model, offload, CPU limits, SSE client and selected physical GPU.
| Engine | Input tokens | Prefill t/s | Decode t/s | Request seconds |
|---|---|---|---|---|
| V100 · expXa a078 · Rust CPU port | 512 | 191.99 (182.14–192.87) | 35.60 (30.27–38.19) | 6.34 (6.01–6.90) |
| V100 · expXa a078 · Rust CPU port | 4096 | 186.92 (185.76–188.91) | 33.18 (31.41–33.41) | 25.82 (25.68–25.89) |
| V100 · PXA4 · Nov 3 matched control | 512 | 209.54 (202.01–214.41) | 37.62 (32.62–38.13) | 5.84 (5.77–6.46) |
| V100 · PXA4 · Nov 3 matched control | 4096 | 202.79 (186.15–206.15) | 33.65 (30.11–33.88) | 24.01 (23.66–26.26) |
Kairo showed no observed regression in these checks: all28 synthetic responses and token/MTP counts matched exactly; repeated text median decode rates were4.6–7.5% higher, and65K-input prefill rates differed by−0.47% to+0.22%. Vision, cache edits and cancellation/drain passed. Both runtimes retain the same immediate-EOS failure for raw-retrieval variant1; chat-formatted variants both pass. Shared-host timing noise and the small fixture set limit generalization.
Small synthetic checks and sampled idle/resource guards support this comparison. They do not establish broad language quality or ordinary Discord latency. Earlier November 2 results and the original Titan/offload study remain below and in all downloads.
Matched engines on the current V100
The follow-up holds GPU 0, 40 CPU expert banks, 8/20 threads, 8,192 context and Q8 KV fixed. All three temporary engine clones use the same extra residency and SSE client. During those November 2 trials, the original Rust process and production routes remained unchanged.
PXA4 recorded the lowest median request time at both prompt lengths in these matched trials. At 4,096 input tokens: current-engine clone 36.77s, PXA4 25.16s, expXa 49.42s. Long-prompt decode ranges overlap; the main gain is prefill.
| Configuration | Prompt tokens | Prefill t/s | Decode t/s | Request s |
|---|---|---|---|---|
| V100 · current live, refreshed | 512 | 123.24 | 32.94 | 8.10 |
| V100 · current live, refreshed | 4096 | 125.61 | 31.11 | 36.56 |
| V100 · current engine clone | 512 | 108.57 | 33.13 | 8.95 |
| V100 · current engine clone | 4096 | 128.13 | 30.10 | 36.77 |
| V100 · PXA4, same settings | 512 | 206.85 | 37.42 | 5.85 |
| V100 · PXA4, same settings | 4096 | 199.57 | 29.30 | 25.16 |
| V100 · expXa, same settings | 512 | 79.85 | 31.94 | 10.42 |
| V100 · expXa, same settings | 4096 | 90.73 | 29.21 | 49.42 |
All matched clones place about 21% of weight buffers on GPU, with unchanged offload. Accepted clone runs show no production work in the sampled observations and retain at least 512 MiB free VRAM. This is a backend throughput test, not ordinary Discord latency or a full language-quality evaluation. The complete original Titan comparison remains below.
The comparison
01 / MeasurementsChange the prompt length and metric. Each bar summarizes only the recorded repetitions for that arm.
Bars show the median. Dots mark individual repetitions. Unavailable or unsupported arms have no bar; -- never means measured zero.
Every recorded run
02 / Raw numbers| Arm | Status | Prompt target | Repeat | Prompt n | Output n | Prefill t/s | Decode t/s | Prompt ms | Decode ms | Request s |
|---|
Compatibility & configuration
03 / What actually ranAn engine’s launch status is part of the result. A forced Overdrive switch is an activation probe; it is not proof that a matching recipe ran.
How to read these results
04 / Protocol & limitsOne fixed workload
Every measured arm uses the same original GGUF weights. Check the model identity above and the settings attached to each arm before comparing rates. No engine earns a result by substituting a different quantization.
The two prompt targets are 512 and 4,096 tokens. The protocol requests 128 output tokens; consult each run’s actual token counts. Forced output length measures execution, and does not test answer quality.
The shared V100 baseline and dedicated Titan Xp runs differ in hardware, CPU allocation, and contention. This is a comparison of observed configurations, not an isolated estimate of an engine’s effect. During the original November 2 benchmark, the live V100 configuration remained unchanged; later activation and tuning status appear above.
Descriptive, not inferential
- Repetition counts vary by phase: inspect each configuration; the latest primary tuning arms use three repetitions per prompt length and the incumbent brackets use two. Actual recorded counts are shown; medians and individual markers are descriptive, with no claim of statistical significance.
- Prefill and decode rates come from engine timing fields. Request seconds are measured wall time and may include overhead outside those engine timers.
- Unsupported, failed, or unmeasured configurations remain visible without being ranked as zero-speed engines.
- Overdrive activation and model architecture support must be established from the evidence, rather than inferred from a requested environment variable.
Receipts, not just bars
05 / EvidenceInspect the machine-readable results and the supplied logs, commands, and compatibility checks.
Downloads become available when the report payload is loaded.