The Radeon AI PRO R9700 has 32 GB of GDDR6 memory, but a running model sees only the portion currently free. The proposed single-card Qwen3.8-27B Q6 profile asks whether that memory tier can support a larger quantization and 32K catalog context without sacrificing useful response time. A separate Q8 candidate uses a shorter 16K context; the difference is a catalog choice, not a tested card limit.
If you have two R9700s, decide whether you need two independent requests or a larger model split across cards. Those need different launchers and measurements. Check both cards’ actual PCIe links, power connections and airflow before considering a pair. Flash-Next with host-memory offload remains separate research, not a recipe shown here.
System setup
PCIe link. AMD lists PCIe 5.0 x16 (AMD specifications). In a PCIe 4.0 slot it runs at PCIe 4.0, still with all 16 lanes. On a riser rated for PCIe 4.0, set the slot to Gen 4 (our recommendation). If the link reads slow under load, see the guide's troubleshooting order.
Loading (calculated). This card's FastLLM model, Qwen3.8-27B UD-Q6_K_M (23.09 GB), loads in at least 3.3 s from a PCIe 4.0 NVMe drive and 42.0 s from a SATA SSD; the drive sets that, not the PCIe generation. A reload from system memory takes 0.37 s at PCIe 5.0 x16 (0.73 s at 4.0).
Engine. FastLLM, in development, uses llama.cpp's Vulkan engine on this card. Bitworks has no lab observation for this card yet. AMD lists this card as RDNA4, gfx1201, in its ROCm GPU specifications. FastLLM's HIP engine currently covers only the RX 7900 XT and XTX.
Resizable BAR. Turn it on; it matters most for the Vulkan engine this card uses. If FastLLM detects it off, it applies its workaround and shows how to turn it on. The workaround is a fallback, not a replacement.
Driver. AMD's driver page for this card offers the AMD Software: PRO Edition driver. We have not tested its install options on this card.
Power and cooling. AMD lists a total board power of 300 W and a minimum power supply recommendation of 750 W (AMD specifications). Plan airflow for sustained load, not short bursts.
Memory headroom. A larger model leaves less of the PC's own memory free while it loads. Plan system memory of at least the model file's size plus what Windows and your other apps use.
Method (as reported): One BetterBench 0.6.0 validation run, 290 successes including warmups, 48 requests per concurrency level.
HEAD seen at retrieval: 8f56157c05eb (not necessarily the commit the author measured)
Reported GPU setup Radeon AI PRO R9700 32GB
GPU count 1
Operating system Linux (Linux x86-64)
Environment Container
Engine vLLM 0.29.0 with Paiton plugin 65K ROCm10 release image
Backend ROCm 10
Model Qwen3.8-27B Unsloth NVFP4 checkpoint via MXFP4 path (Unsloth)
Configured context 65,536 tokens
Actual input input length not reported
Full window tested no
Reported context Concurrent prompt occupancy unreported; separate prefill sweep reached median 47,016.5.
Server slots 8
Topology single
Prefix cache off
Speculative decoding DFlash2
115 tok/s — generation speed (aggregate across clients) · API server · 1 client C1; completed output/common full-workload wall; 48 requests per level; prompt occupancy unreported. This row: configured context: 65,536 tokens · engine: vLLM 0.29.0 + Paiton · speculative decoding: DFlash2
203.2 tok/s — generation speed (aggregate across clients) · API server · 2 clients C2; completed output/common full-workload wall; 48 requests per level; prompt occupancy unreported. This row: configured context: 65,536 tokens · engine: vLLM 0.29.0 + Paiton · speculative decoding: DFlash2
296.5 tok/s — generation speed (aggregate across clients) · API server · 4 clients C4; completed output/common full-workload wall; 48 requests per level; prompt occupancy unreported. This row: configured context: 65,536 tokens · engine: vLLM 0.29.0 + Paiton · speculative decoding: DFlash2
400.7 tok/s — generation speed (aggregate across clients) · API server · 8 clients C8; completed output/common full-workload wall; 48 requests per level; prompt occupancy unreported. This row: configured context: 65,536 tokens · engine: vLLM 0.29.0 + Paiton · speculative decoding: DFlash2
Not reported: model file checksum, CPU, system memory, concurrent prompt occupancy and per-client rates, independent replication on Bitworks hardware
Same Paiton project/author as the original register's long-KV4 entry; do not count as independent corroboration.
Different checkpoint, runtime, sampling and workload from Paiton's earlier 8K matched Radiance comparison; no cross-release speedup percentage follows.
65K is configured capacity, not eight simultaneous full-65K conversations; Linux result is not native Windows.
Method (as reported): Project author reports BetterBench 20-pass runs and a repeated-prefix, 20-turn agent session.
HEAD seen at retrieval: 918be25e10f1 (not necessarily the commit the author measured)
Reported GPU setup Radeon AI PRO R9700
GPU count 1
Operating system Linux (x86-64 Docker ROCm)
Environment Container
Engine stock vLLM plus Paiton plugin
Backend ROCm
Model Qwen3.8-27B project 3-bit mode
Configured context 262,144 tokens
Actual input input length not reported
Full window tested no
Reported context Agent session grew from about 50K to 253K; exact input per metric unknown.
Workers 1
Topology single
173.4 tok/s — generation speed (per request) · 1 client Compared with 174.3 tok/s (baseline prior mode; reported long-kv4); reported difference: other Single-stream decode comparison; cache conditions not fully specified. This row: configured context: 262,144 tokens
478.5 tok/s — generation speed (aggregate across clients) · 8 clients Compared with 462.4 tok/s (prior mode vs long-kv4); reported difference: other Aggregate across eight requests; not single-client decode speed. This row: configured context: 262,144 tokens
204 s — task time (scope not stated) · agent session · 1 client Compared with 1237 s (baseline prior mode; reported warm repeated-prefix agent task); reported difference: prefix cache Total for a 20-turn warm repeated-prefix session, not one request; benefit does not isolate decode. This row: configured context: 262,144 tokens · prefix cache: on
569,878 tokens — reusable prefix cache (server total) (server-wide) Total reusable prefix tokens, not a per-request window. This row: configured context: 262,144 tokens · prefix cache: on
Not reported: engine build, CPU, system memory, container digest, model digest, independent replication, Bitworks workload eviction behavior
Warm prefix reuse explains task-time change; it is not decode acceleration.
Cold first read remains expensive; short or cold prompts can be slower.
Method (as reported): BetterBench 0.4.0 on two passthrough cards with shared emulated PCIe switch and repaired RCCL/P2P path.
HEAD seen at retrieval: e2a38bf4ab5d (not necessarily the commit the author measured)
Reported GPU setup Radeon AI PRO R9700
GPU count 2
Operating system Linux (Proxmox 9.2 VM; Debian 13 guest)
Environment VM guest
Engine vLLM plus DFlash2 FP8 and Radiance vLLM 0.28; BetterBench 0.4.0
Backend ROCm
Model Qwen3.8-27B MXFP4-W4A8
Actual input input length not reported
Reported context Source spans multiple depths; per-metric context differs or is unreported.
Workers 1
Topology dual-split (PCIe 5.0 x16 each; VM P2P path)
CPU EPYC 7543
PCIe PCIe 5.0 x16 each with VM P2P plumbing
159.7 tok/s — other (per request) · 1 client Compared with 106.2 tok/s (baseline FP8+DFlash2; reported MXFP4-W4A8+DFlash2-FP8); changed between the two runs: quantisation, engine BetterBench weighted single-stream generation; weight and kernel paths differ. This row: quantisation: MXFP4-W4A8 · engine: vLLM plus DFlash2 FP8 and Radiance
495 tok/s — generation speed (aggregate across clients) · 8 clients Aggregate across eight simultaneous streams; not per-request generation. This row: quantisation: MXFP4-W4A8
≈5153 tok/s — prompt speed (per request) · 1 client Approximate input depth around 16K; not exact occupancy. This row: quantisation: MXFP4-W4A8
Not reported: configured context, system memory, model artifact digest, bare-metal behavior, Windows behavior, per-request context in summary
FP8 and MXFP4 comparison changes weights, kernels, and runtime path.
Reported total KV cache capacity is not per-request tested context.
VM-specific P2P workaround is not a general installer instruction.
Model Qwen3.8-27B abihsoro AWQ INT4 W4A16 group-128 symmetric (abihsoro)
Configured context 512 tokens
Actual input 32 tokens
Output 128 tokens
Full window tested no
Reported context Short 32-input/128-output workload; not long-context agents.
Server slots 32
Topology dual-split (Reported arm TP2 RCCL on two GPUs; baseline TP1 on one GPU.)
Prefix cache off
13.65 tok/s — generation speed (aggregate across clients) · API server · 1 client · input 32 tokens Compared with 21.31 tok/s (TP1 one R9700 vs TP2 RCCL two R9700); changed between the two runs: GPU count, split mode C1; 8 requests/arm; completed output/common full-workload wall; TP2 RCCL two GPUs. This row: configured context: 512 tokens · topology: dual-split · cards: 2
69.52 tok/s — generation speed (aggregate across clients) · API server · 8 clients · input 32 tokens Compared with 66.65 tok/s (TP1 one R9700 vs TP2 RCCL two R9700); changed between the two runs: GPU count, split mode C8; 16 requests/arm; completed output/common full-workload wall; TP2 RCCL two GPUs. This row: configured context: 512 tokens · topology: dual-split · cards: 2
179.73 tok/s — generation speed (aggregate across clients) · API server · 32 clients · input 32 tokens Compared with 170.48 tok/s (TP1 one R9700 vs TP2 RCCL two R9700); changed between the two runs: GPU count, split mode C32; 64 requests/arm; completed output/common full-workload wall; TP2 RCCL two GPUs. This row: configured context: 512 tokens · topology: dual-split · cards: 2
1067.01 ms — time to first token (per request) · API server · 32 clients · input 32 tokens Compared with 780.45 ms (TP1 one R9700 vs TP2 RCCL two R9700); changed between the two runs: GPU count, split mode C32 mean TTFT across 64 requests/arm; not p95; TP2 RCCL two GPUs. This row: configured context: 512 tokens · topology: dual-split · cards: 2
Not reported: CPU, system memory, exact reference-run binary digest and full machine topology in this summary, performance at useful agent contexts or with mixed long prompts, performance on RX 7900 XT, RX 7900 XTX or current RC2 runtime
Fresh TP2 qualification of final RC2 vLLM/ROCm 10 commits is pending after a power interruption. These are prior reference measurements, not a current RC2 benchmark.
Only dual-R9700 gfx1201 was historically end-to-end exercised; gfx1100 is build-selectable but unvalidated. No dual-XT recipe or performance claim follows.
512 configured context and 32-token inputs do not represent long-context subagent workloads; TP2 gains only 5.4% aggregate at C32 and has higher mean TTFT in this row.
Custom native-Windows fork/transport is not FastLLM's pinned llama.cpp Vulkan lane.
IQ4_XS target; Q8_0 DFlash2 draft · 1 card · Lucebox HIP and separate llama.cpp ROCm 7.2.4 · 106.6 tok/s generation (aggregate) · by Umberto Breglia / Lucebox on Blog
Reported on Blog by Umberto Breglia / Lucebox Original report
R9700 Lucebox continuous-batching comparison
Method (as reported): Ten synchronized HumanEval waves/concurrency, median of three fresh-server repetitions after warmup; output/common full request wall.
Reported GPU setup Radeon AI PRO R9700 32GB
GPU count 1
Operating system OS not reported — GPU-only evidence
Engine Lucebox HIP and separate llama.cpp
Backend ROCm 7.2.4
Model Qwen3.8-27B IQ4_XS target; Q8_0 DFlash2 draft
Configured context 16,384 tokens
Actual input input length not reported
Output 256 tokens
Full window tested no
Reported context HumanEval short prompts; separate retrieval depth 1,046–15,695.
Server slots 6
Topology single
Prefix cache off
Speculative decoding DFlash2
106.6 tok/s — generation speed (aggregate across clients) · API server · 1 client Compared with 75.2 tok/s (Separate llama.cpp run vs Lucebox HIP); reported difference: engine C1 HumanEval; total completed output / common full-request wall, not isolated decode. This row: configured context: 16,384 tokens · engine: Lucebox HIP · speculative decoding: DFlash2
300.9 tok/s — generation speed (aggregate across clients) · API server · 5 clients Compared with 193 tok/s (Separate llama.cpp run vs Lucebox HIP); reported difference: engine C5 HumanEval; total completed output / common full-request wall, not per-client decode. This row: configured context: 16,384 tokens · engine: Lucebox HIP · speculative decoding: DFlash2
6.2 tok/s — generation speed (aggregate across clients) · API server · 5 clients · input 15,695 tokens C5 separate 15,695-token retrieval prompt; 128 output/request; median TTFT 100.0 s. This row: configured context: 16,384 tokens · engine: Lucebox HIP · speculative decoding: DFlash2
Not reported: operating system, engine build, CPU, system memory, exact OS distribution and model artifact digest, raw requests/prompt fixtures (author says retained on measurement host), performance under mixed/random arrivals and cancellation
ROCm R9700 result, not native Windows or RX 7900 XT guidance.
Lucebox and llama.cpp were separate runs; no single-resident paired process.
Short coding-prompt scaling does not predict long-context throughput; the long-prompt result is a separate workload.
Discussion
Share setups, questions and your own results. Posts are reviewed before they appear. Replies from Bitworks staff are marked.