Kimi K3 infernce optimization on B300

https://lightseek.org/blog/kimi-k3-optimization-gb300-part-i.html

The post from TokenSpeed explains how LightSeek optimized Kimi K3 inference on eight NVIDIA GB300 GPUs within one NVLink domain, using long, multi-turn coding-agent workloads.
Main takeaways:

• EAGLE3 speculative decoding improves the throughput/latency tradeoff. It generates multiple candidate tokens per target-model pass, increasing total throughput without sacrificing as much per-user speed. At an interpolated 50 tokens/sec per user, it reaches about 459K tokens/min/GPU.
• LatentMoE projection sharding saves ~7.7 GiB per GPU. Splitting the down- and up-projection weights across the eight GPUs eliminates replicated parameters and computation. Communication is carefully fused with projection and reduction steps, so latency generally improves too.
• Replay SSM dramatically reduces KDA verification memory. Instead of snapshotting the full recurrent state for every speculative position, the system stores compact inputs and reconstructs the accepted state afterward. Workspace falls from 16.7 GiB to 0.4 GiB per GPU—a 97.8% reduction—with no observed latency penalty.
• Small-operation overhead matters. Because Kimi K3 repeats operations across 93 layers, LightSeek routes projections to different kernels depending on tensor shape and fuses KDA, MLA, AttnRes, cache-writing, and metadata operations.
• These memory savings are theoretically equivalent to FP8 cache space for roughly 1.8 million additional tokens per GPU, although that is not a guaranteed allocatable capacity.
  Important caveats: the tests cover TP8 inside a single NVLink domain, not expert parallelism, prefill/decode disaggregation, or cross-rack networking. Many reported kernel timings are isolated ablations and must not be added together as an end-to-end speedup. Parts II and III will address expert parallelism and distributed production serving.