AMD RX 6800 XT Kernel Mod Boosts MoE Speed to 1770 Tokens Per Second

Developer Stormrage34 has released TurboQuant-HIP v0.3.0, a fork of llama.cpp optimized for AMD GPUs. The update significantly boosts the performance of Mixture-of-Experts (MoE) model inference on the Radeon RX 6800 XT.

MoE prefill speed jumps 3.7x

MoE prefill speed jumped from approximately 480 tokens per second to 1770 tokens per second. A new BF16-based IQ4_XS dequantization kernel runs 13 times faster than the previous implementation. Asynchronous pipeline scheduling reduces kernel launch overhead by 31 percent. An experimental LDS double-buffered matrix multiplication kernel enables parallel weight loading and DP4A compute, though it is opt-in and not yet production-ready due to a bank conflict issue with symmetric tile sizes.

The update is available globally as of May 13, 2025.

The optimization targets the RX 6800 XT, a GPU originally designed for gaming, and demonstrates the potential for consumer hardware in AI workloads. Stormrage34 has not confirmed a timeline for resolving the LDS bank conflict issue.