AWS Details H200 Kernel Optimizations for MoE Inference
Three changes target matrix utilization, FP8 backend selection and repeated compilation, but the reported gains come from AWS tests on a representative workload.
Edited by Tyronne Panaino
AWS published an engineering account on October 13 describing three changes to fused Mixture-of-Experts inference on NVIDIA H200 GPUs. The work targets low hardware utilization, an inefficient FP8 backend choice and repeated just-in-time compilation. It matters to teams serving sparse models through vLLM or Amazon SageMaker AI because two changes are upstream in vLLM and one is available in AWS's Large Model Inference container.
The AWS engineering article reports gains of 2–4 percent in one optimization, 10 percent in another and 9 percent for time to first token in a third test. Those figures are useful implementation evidence, but they remain AWS-run results from a representative MoE workload rather than a general benchmark across models, accelerators and traffic patterns.
Three bottlenecks require different fixes
A Mixture-of-Experts model activates only part of its capacity for each token. In a fused implementation, token routing, data movement and expert calculations are combined into one GPU operation instead of running as separate loops. That can reduce overhead, but it also makes performance sensitive to matrix shapes, quantization backends and compiler behavior.
AWS's three changes operate at different layers. The first reorganizes matrix multiplication to fit Hopper Tensor Core layout constraints. The second changes which backend handles FP8 tensor-quantized MoE work. The third prevents Triton from recompiling a kernel when a variable token count changes. Treating them separately helps explain why there is no single universal MoE speed switch.
Swap-AB fills more of the Tensor Core layout
During token generation, a server can process only one new token for each active sequence. That creates a thin token dimension which may be smaller than the 64-row layout expected by Hopper's WGMMA instruction path. AWS says idle execution lanes result when the small dimension occupies the fixed leading operand slot.
The Swap-AB optimization transposes the mathematical operation before it reaches the low-level assembly layer, moving the much larger expert dimension into that leading position. AWS reports a 2–4 percent improvement in end-to-end output throughput across large-scale MoE models on H200. The change was contributed to vLLM and is available from version 0.24.0.
This is a layout optimization, not a change to model weights or routing policy. Its practical value depends on whether a deployment has the same thin decode shapes and Hopper-specific bottleneck that motivated the work.
FP8 routing can avoid an inefficient default
AWS also found that Triton outperformed the default FlashInfer plus CUTLASS route for FP8 tensor-quantized fused MoE models on Hopper. The explanation given is that Triton can fuse scaling, matrix multiplication and expert aggregation while adapting its scheduling to small, irregular token batches.
Changing the routing policy required configuration rather than new kernel code. AWS reports a 10 percent end-to-end throughput improvement on H200, and says the route is available in version 28 of its Large Model Inference container. Teams using another quantization format, backend, concurrency profile or accelerator should not assume the same result without measuring their own serving path.
Removing recompilation shortens the first response delay
The third change addresses Triton's tendency to specialize integer parameters. A previously unseen value can trigger a fresh JIT compilation, which is expensive when sequence lengths and batch sizes vary. AWS identified a fused rotary-position and FP8-quantization kernel that recompiled for new token counts.
Adding `do_not_specialize` annotations to variable-length parameters removed those recompilations. AWS reports a 9 percent improvement in time to first token on H200 with eight-way tensor parallelism. The optimization was upstreamed to vLLM and is available from version 0.22.0.
In a multi-GPU deployment, avoiding one worker's recompilation pause can also reduce time spent waiting across the synchronized group. The reported result still belongs to the tested eight-GPU setup; it does not establish the same gain for every parallelism strategy.
The next checkpoint is workload-level reproduction
The release is most actionable for operators already using H200, vLLM or the AWS LMI container. They can compare the documented versions and routing choice against a pinned baseline, then measure throughput, time to first token, tail latency, memory use and output correctness on their own traffic.
The source does not identify the representative model or provide a complete benchmark protocol, concurrency sweep, cost comparison or independent reproduction. It also does not show whether the three headline gains add together in one deployment. The supported conclusion is therefore narrow: AWS has published and productized three concrete optimizations, while their combined value remains model- and workload-specific.
Status
Learning. Internal confidence is medium because AWS documents the implementation and its own measurements, but no independent reproduction or cross-platform comparison was fetched for this article.
Sources
Update note: Last reviewed 2026-10-14. We will revise this post if AWS or the vLLM project publishes fuller benchmark conditions, additional hardware results or independent reproduction evidence.
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.