AMD Radeon AI PRO R9700 32GB

Reported on GitHub by bkvargyas Original report

Dual R9700 vLLM Proxmox recipe and benchmarks

Method (as reported): BetterBench 0.4.0 on two passthrough cards with shared emulated PCIe switch and repaired RCCL/P2P path.

HEAD seen at retrieval: e2a38bf4ab5d (not necessarily the commit the author measured)

• Reported GPU setup Radeon AI PRO R9700
• GPU count 2
• Operating system Linux (Proxmox 9.2 VM; Debian 13 guest)
• Environment VM guest
• Engine vLLM plus DFlash2 FP8 and Radiance vLLM 0.28; BetterBench 0.4.0
• Backend ROCm
• Model Qwen3.8-27B MXFP4-W4A8
• Actual input input length not reported
• Reported context Source spans multiple depths; per-metric context differs or is unreported.
• Workers 1
• Topology dual-split (PCIe 5.0 x16 each; VM P2P path)
• CPU EPYC 7543
• PCIe PCIe 5.0 x16 each with VM P2P plumbing

• 159.7 tok/s — other (per request) · 1 client
  Compared with 106.2 tok/s (baseline FP8+DFlash2; reported MXFP4-W4A8+DFlash2-FP8); changed between the two runs: quantisation, engine
  BetterBench weighted single-stream generation; weight and kernel paths differ.
  This row: quantisation: MXFP4-W4A8 · engine: vLLM plus DFlash2 FP8 and Radiance
• 495 tok/s — generation speed (aggregate across clients) · 8 clients
  Aggregate across eight simultaneous streams; not per-request generation.
  This row: quantisation: MXFP4-W4A8
• ≈5153 tok/s — prompt speed (per request) · 1 client
  Approximate input depth around 16K; not exact occupancy.
  This row: quantisation: MXFP4-W4A8

Not reported: configured context, system memory, model artifact digest, bare-metal behavior, Windows behavior, per-request context in summary

• FP8 and MXFP4 comparison changes weights, kernels, and runtime path.
• Reported total KV cache capacity is not per-request tested context.
• VM-specific P2P workaround is not a general installer instruction.

Setup guide (external)

Reported on GitHub by charlie12345 · measured 2026-08-26 Original report

Native Windows dual R9700 custom-vLLM reference concurrency

Method (as reported): OpenAI-compatible serving; warmup wave per level, forced 128 output, no prefix cache; TP1 one GPU vs TP2 RCCL two GPUs.

HEAD seen at retrieval: 9c9087be48a5 (not necessarily the commit the author measured)

• Reported GPU setup 2 × Radeon AI PRO R9700 32GB
• GPU count 2
• Operating system Windows (Windows 11)
• Engine custom native-Windows vLLM fork prior reference build; current RC2 TP2 requalification pending
• Backend ROCm Direct-RCCL
• Model Qwen3.8-27B abihsoro AWQ INT4 W4A16 group-128 symmetric (abihsoro)
• Configured context 512 tokens
• Actual input 32 tokens
• Output 128 tokens
• Full window tested no
• Reported context Short 32-input/128-output workload; not long-context agents.
• Server slots 32
• Topology dual-split (Reported arm TP2 RCCL on two GPUs; baseline TP1 on one GPU.)
• Prefix cache off

• 13.65 tok/s — generation speed (aggregate across clients) · API server · 1 client · input 32 tokens
  Compared with 21.31 tok/s (TP1 one R9700 vs TP2 RCCL two R9700); changed between the two runs: GPU count, split mode
  C1; 8 requests/arm; completed output/common full-workload wall; TP2 RCCL two GPUs.
  This row: configured context: 512 tokens · topology: dual-split · cards: 2
• 69.52 tok/s — generation speed (aggregate across clients) · API server · 8 clients · input 32 tokens
  Compared with 66.65 tok/s (TP1 one R9700 vs TP2 RCCL two R9700); changed between the two runs: GPU count, split mode
  C8; 16 requests/arm; completed output/common full-workload wall; TP2 RCCL two GPUs.
  This row: configured context: 512 tokens · topology: dual-split · cards: 2
• 179.73 tok/s — generation speed (aggregate across clients) · API server · 32 clients · input 32 tokens
  Compared with 170.48 tok/s (TP1 one R9700 vs TP2 RCCL two R9700); changed between the two runs: GPU count, split mode
  C32; 64 requests/arm; completed output/common full-workload wall; TP2 RCCL two GPUs.
  This row: configured context: 512 tokens · topology: dual-split · cards: 2
• 1067.01 ms — time to first token (per request) · API server · 32 clients · input 32 tokens
  Compared with 780.45 ms (TP1 one R9700 vs TP2 RCCL two R9700); changed between the two runs: GPU count, split mode
  C32 mean TTFT across 64 requests/arm; not p95; TP2 RCCL two GPUs.
  This row: configured context: 512 tokens · topology: dual-split · cards: 2

Not reported: CPU, system memory, exact reference-run binary digest and full machine topology in this summary, performance at useful agent contexts or with mixed long prompts, performance on RX 7900 XT, RX 7900 XTX or current RC2 runtime

• Fresh TP2 qualification of final RC2 vLLM/ROCm 10 commits is pending after a power interruption. These are prior reference measurements, not a current RC2 benchmark.
• Only dual-R9700 gfx1201 was historically end-to-end exercised; gfx1100 is build-selectable but unvalidated. No dual-XT recipe or performance claim follows.
• 512 configured context and 32-token inputs do not represent long-context subagent workloads; TP2 gains only 5.4% aggregate at C32 and has higher mean TTFT in this row.
• Custom native-Windows fork/transport is not FastLLM's pinned llama.cpp Vulkan lane.

Setup guide (external)

Related link (external)

Reported on Blog by Umberto Breglia / Lucebox Original report

R9700 Lucebox continuous-batching comparison

Method (as reported): Ten synchronized HumanEval waves/concurrency, median of three fresh-server repetitions after warmup; output/common full request wall.

• Reported GPU setup Radeon AI PRO R9700 32GB
• GPU count 1
• Operating system OS not reported — GPU-only evidence
• Engine Lucebox HIP and separate llama.cpp
• Backend ROCm 7.2.4
• Model Qwen3.8-27B IQ4_XS target; Q8_0 DFlash2 draft
• Configured context 16,384 tokens
• Actual input input length not reported
• Output 256 tokens
• Full window tested no
• Reported context HumanEval short prompts; separate retrieval depth 1,046–15,695.
• Server slots 6
• Topology single
• Prefix cache off
• Speculative decoding DFlash2

• 106.6 tok/s — generation speed (aggregate across clients) · API server · 1 client
  Compared with 75.2 tok/s (Separate llama.cpp run vs Lucebox HIP); reported difference: engine
  C1 HumanEval; total completed output / common full-request wall, not isolated decode.
  This row: configured context: 16,384 tokens · engine: Lucebox HIP · speculative decoding: DFlash2
• 300.9 tok/s — generation speed (aggregate across clients) · API server · 5 clients
  Compared with 193 tok/s (Separate llama.cpp run vs Lucebox HIP); reported difference: engine
  C5 HumanEval; total completed output / common full-request wall, not per-client decode.
  This row: configured context: 16,384 tokens · engine: Lucebox HIP · speculative decoding: DFlash2
• 6.2 tok/s — generation speed (aggregate across clients) · API server · 5 clients · input 15,695 tokens
  C5 separate 15,695-token retrieval prompt; 128 output/request; median TTFT 100.0 s.
  This row: configured context: 16,384 tokens · engine: Lucebox HIP · speculative decoding: DFlash2

Not reported: operating system, engine build, CPU, system memory, exact OS distribution and model artifact digest, raw requests/prompt fixtures (author says retained on measurement host), performance under mixed/random arrivals and cancellation

• ROCm R9700 result, not native Windows or RX 7900 XT guidance.
• Lucebox and llama.cpp were separate runs; no single-resident paired process.
• Short coding-prompt scaling does not predict long-context throughput; the long-prompt result is a separate workload.

Setup guide (external)

Related link (external)

Reported on GitHub by JohnTDI-cpu Original report

Direct P2P AllReduce on dual R9700

Method (as reported): llama-bench -sm tensor -fa on -n 128 -r 5; author compared perplexity; prefill remained on stock butterfly path.

HEAD seen at retrieval: f6f66106f2a9 (not necessarily the commit the author measured)

• Reported GPU setup Radeon AI PRO R9700
• GPU count 2
• Operating system Linux (ROCm 7.2.4)
• Engine llama.cpp HIP tensor plus direct-P2P patch latest master at author test; commit unknown
• Backend ROCm
• Model Qwen3-27B Q8_0
• Actual input input length not reported
• Simultaneous clients 1
• Workers 1
• Topology dual-split (tensor split; peer access available, link layout unknown)
• PCIe peer access reported; layout unknown

• 30.43 tok/s — generation speed (per request) · Synthetic benchmark (not API serving) · 1 client
  Compared with 28.3 tok/s (baseline stock tensor; reported direct-P2P patch); reported difference: engine build
  Synthetic llama-bench tg128; five repetitions, not API serving.
  This row: topology: dual-split

Not reported: configured context, actual input length, CPU, system memory, tested engine commit, artifact digest, PCIe topology, API concurrency

• Model is Qwen3-27B, not Qwen3.8-27B.
• One-machine synthetic decode does not establish API serving throughput.
• Patch depends on peer access and was not tested on Windows.

Setup guide (external)