#Claude #Codex # I Tried Running a Local LLM on a Ryzen Zen3 GPU (Qwen3-8B Heretic: Performance Comparison and Analysis of CPU vs GPU) Keep going, AMD, keep going, Vulkan. I thought it wouldn’t amount to much since it’s a gaming GPU—and I was right. Apparently, memory bandwidth is the bottleneck (due to small VRAM, relying on main memory; looking forward to Mac M5 Studio). AMD’s latest is Zen6, so this is a CPU three generations old (with integrated GPU). ---- # Qwen3-8B Heretic: Performance Comparison and Analysis of CPU vs GPU Measurement Date: 2026-11-02 / Target: Ubuntu Machine ## Conclusion - **The GPU (integrated Radeon) is about 2.5x faster than the CPU during prompt loading.** - **Text generation is only 1.1–1.2x faster on the GPU.** Generation speed is capped by memory transfer rates, as the integrated GPU shares the same memory as the CPU. - When using long instruction prompts repeatedly—as with OpenClaw—the benefit of **prompt caching** outweighs GPU acceleration; subsequent responses dropped from ~7 minutes to ~1 minute. ## Measurement Environment | Item | Details | |---|---| | CPU | AMD Ryzen 7 7730U (8 cores, 16 threads, Zen 3) | | GPU | Integrated AMD Radeon Graphics (Barcelo, equivalent to Vega 8) | | Memory | 30GB (Specified as DDR4-3200 dual-channel; actual configuration unverified) | | GPU Memory | Dedicated VRAM: 512MB + Shared Memory (GTT): ~15.4GB. LM Studio can use 10.59GiB. | | OS / Driver | Ubuntu 24.04.5 LTS, Mesa 25.2.8 (RADV) | | LM Studio | Headless version (llmster), llama.cpp 2.66.0 | | CPU Runtime | `llama.cpp-linux-x86_64-avx2` | | GPU Runtime | `llama.cpp-linux-x86_64-vulkan-avx2` (all layers on GPU, `--gpu max`) | | Model | DreamFast/qwen3-8b-heretic Q4_K_M (5.03GB) | | Loading Settings | Context length: 32768, Concurrent requests: 1 | ## Measurement Method - Sent identical prompts via LM Studio’s API (`/api/v0/chat/completions`) and compared CPU vs GPU performance. - Each condition was executed twice. For the second run, prompts were reused from cache; thus, **loading times use values from the first run**. - Generation speed is measured using LM Studio’s reported `tokens_per_second`. - Pre-generation reasoning was disabled using `/no_think`, and `temperature` was set to 0. - After GPU measurements, the system cooled for 2 minutes before CPU measurements. - Scripts and logs are located at `~/bench/bench.py`, `~/bench/run.sh`, and `~/bench/result.log` on the Ubuntu machine. ## Results ### Summary Comparison | Category | Metric | CPU | GPU | Ratio | |---|---|---|---|---| | Short query (24 tokens) generating 256 tokens | Generation Speed | 7.3 tokens/sec | 8.2 tokens/sec | 1.13x | | | Total Time | 35.7 sec | 31.4 sec | | | Prompt: 921 tokens | Loading Speed | 30.5 tokens/sec | 76.3 tokens/sec | **2.5x** | | | Time to First Token | 30.2 sec | 12.1 sec | | | | Generation Speed | 7.0 tokens/sec | 8.0 tokens/sec | 1.13x | | Prompt: 3,621 tokens | Loading Speed | 35.4 tokens/sec | 87.5 tokens/sec | **2.5x** | | | Time to First Token | 102.3 sec | 41.4 sec | | | | Generation Speed | 5.9 tokens/sec | 7.2 tokens/sec | 1.2x | ### Detailed Measurements | Method | Description | Run | Prompt Tokens | Generated Tokens | Total Time | Time to First Token | Generation Speed | |---|---|---|---|---|---|---|---| | GPU | Short query | 1 | 24 | 256 | 31.4 sec | 0.50 sec | 8.26 | | GPU | Short query | 2 | 24 | 256 | 31.2 sec | 0.12 sec | 8.23 | | GPU | Prompt ~1,000 tokens | 1 | 921 | 32 | 16.0 sec | 12.07 sec | 7.97 | | GPU | Prompt ~1,000 tokens (cached) | 2 | 921 | 32 | 4.0 sec | 0.14 sec | 7.98 | | GPU | Prompt ~4,000 tokens | 1 | 3,621 | 32 | 45.7 sec | 41.38 sec | 7.18 || GPU | Prompt ~4,000 | 2 (cached) | 3,621 | 32 | 4.5 sec | 0.14 sec | 7.22 | | CPU | Short question | 1 | 24 | 256 | 35.7 sec | 0.73 sec | 7.29 | | CPU | Short question | 2 | 24 | 241 | 32.8 sec | 0.14 sec | 7.36 | | CPU | Prompt ~1,000 | 1 | 921 | 32 | 34.7 sec | 30.21 sec | 7.03 | | CPU | Prompt ~1,000 | 2 (cached) | 921 | 32 | 4.6 sec | 0.15 sec | 6.96 | | CPU | Prompt ~4,000 | 1 | 3,621 | 32 | 107.5 sec | 102.30 sec | 5.94 | | CPU | Prompt ~4,000 | 2 (cached) | 3,621 | 32 | 5.4 sec | 0.17 sec | 5.96 | Units for prompt and generation are tokens; generation speed is measured in tokens/sec. ### Correction Previously, we stated that generation speed on the GPU was approximately 2.2 times faster, but this was incorrect. The CPU value of 3.86 tokens/sec was measured when processing a prompt of approximately 14,000 tokens on bot2. Since longer prompts result in slower generation, comparing it to the short-question scenario was misleading. When compared under identical conditions, the generation speed difference is only 1.1 to 1.2 times. ## Why Using a GPU Doesn’t Result in Much Faster Performance ### 1. Generation is Limited by Memory Transfer Speed Each time an LLM generates one token, it must read all model weights from memory. The upper limit of generation speed is roughly determined by: “Memory transfer speed ÷ Model size.” | Item | Value | |---|---| | Memory transfer speed (theoretical) | ~51 GB/s (dual-channel DDR4-3200) | | Amount read per token | ~5 GB (entire model) | | Theoretical upper limit of generation speed | 51 ÷ 5.03 ≒ **~10 tokens/sec** | | Measured (GPU) | 8.2 tokens/sec (~80% of limit) | | Measured (CPU) | 7.3 tokens/sec (~70% of limit) | The integrated GPU shares the same main memory as the CPU, so its upper limit is nearly identical to that of the CPU. Even with a GPU, there is only about a 10–20% potential improvement toward reaching this limit. Discrete GPUs have dedicated high-speed memory, significantly raising this upper bound. | GPU | Memory Transfer Speed | Ratio Compared to This Ubuntu Machine | |---|---|---| | Integrated GPU in this Ubuntu machine | ~51 GB/s (shared with CPU) | 1x | | RTX 4060 | ~272 GB/s | ~5x | | RTX 4090 | ~1,008 GB/s | ~20x | ### 2. Loading Depends on Computational Power, but the Integrated GPU Has Limited Compute Capacity During prompt loading, many tokens are processed in bulk, so speed is determined more by computational power than memory bandwidth—the area where GPUs typically excel. In practice, we observed a 2.5x improvement here. However, this GPU has the following constraints, limiting performance to just 2.5x: | Constraint | Details | |---|---| | Few compute units | Only 8 units; estimated single-precision performance ~2 TFLOPS—just a fraction of gaming GPUs | | No dedicated matrix运算 circuits | Lacks specialized AI hardware like NVIDIA’s Tensor Cores | | ROCm not supported | AMD’s GPU computing platform ROCm does not officially support this integrated GPU; it runs via generic Vulkan | Processing one token in an 8B model requires approximately 16 GFLOPs of computation. With ~2 TFLOPS, the theoretical upper limit is ~120 tokens/sec. The measured values of 76–88 tokens/sec represent about 60–70% of this theoretical maximum. ### 3. Longer Prompts Result in Slower Performance As context length increases, each token must reference more prior tokens, slowing down both loading and generation. | Condition | Loading Speed | Generation Speed | |---|---|---| | GPU, prompt ~1,000 tokens | 76 tokens/sec | 8.0 tokens/sec | | GPU, prompt ~3,600 tokens | 88 tokens/sec | 7.2 tokens/sec | | GPU, OpenClaw (15,686 tokens), first 2,048 tokens | 78.7 tokens/sec | — | | GPU, OpenClaw (15,686 tokens), overall average | 44.1 tokens/sec | 5.1 tokens/sec | | CPU, OpenClaw (14,021 tokens) | ~19.5 tokens/sec | 3.86 tokens/sec |For up to 3,600 tokens, processing larger batches simultaneously actually improved loading speed. Beyond 15,000 tokens, the rate dropped to as low as 46 tokens/second at the end. ## Cache Validation with OpenClaw (bot2 Mei) Two consecutive requests were sent using Qwen3 running on GPU within a test session. | Item | First Request | Second Request | |---|---|---| | Prompt sent by OpenClaw | 15,686 tokens | 15,744 tokens | | Actual tokens loaded | 15,686 (all) | **135 (only newly added)** | | Loading time | 356.1 seconds | **6.1 seconds** | | Generation | 247 tokens, 48.1 seconds (5.11 tokens/sec) | 249 tokens, 48.5 seconds (5.11 tokens/sec) | | Total | **410 seconds** | **58 seconds** | - OpenClaw consistently sends the same instruction text at the beginning of each prompt, so 98% of the content from the first request was reused in the second. - Most of the waiting time in the second request was due to approximately 240 tokens of internal reasoning. Even with `--thinking off` specified, reasoning did not stop. If reasoning could be disabled, total time could be reduced to around 10 seconds. - Cache does not expire based on time. It is cleared only when the model is unloaded, another process interrupts (due to a limit of one concurrent process), or the conversation is compressed. ## Options for Improving Speed Since memory transfer speed cannot be changed, reducing the amount loaded per token is effective. | Method | Expected Benefit | Notes | |---|---|---| | Leverage prompt caching | Subsequent requests load in just a few seconds (validated) | Cache can be cleared by interruptions or compression | | Disable Qwen3 reasoning | Reduce per-request time by ~48 seconds | May slightly reduce answer quality | | Switch to MoE Qwen3-30B-A3B | Uses only ~3B weights per token; generation expected to be 2–3x faster with improved reasoning | Model size is ~18GB, exceeding GPU’s 10.6GiB capacity—some processing occurs on CPU. Unverified whether a censored version is available | | Use smaller model (Qwen3-4B) | Generation speed ~2x faster | Reduced reasoning capability | | Use speculative decoding | Use Qwen3-0.6B as draft model; generation expected to be 1.3–2x faster | Effectiveness varies by content | | Further quantization (e.g., Q3) | ~10–20% improvement | Reduced accuracy | | Shorten instruction text | Reduces initial load and reload time after cache loss | Requires sacrificing functionality | Except for prompt caching, all benefits listed above are estimates and have not been measured empirically. ## Current Status (as of 2026-11-03) | Item | Status | |---|---| | LM Studio | Stopped; auto-start disabled | | Selected runtime | Vulkan (GPU) | | bot2 (Mei) | gpt-5.4-mini | | Transfer port for bot2 | Stopped; auto-start disabled | | Qwen3 configuration for bot2 | `~/.openclaw-bot2/openclaw.json.bak-qwen3-active-*` |