6 articles tagged
P-EAGLE, DFlash, and DSpark bring parallel drafting to vLLM in 2026, generating a whole block of draft tokens in one pass for up to 1.69x higher throughput than EAGLE-3. How they work and how to serve them.
Kimi K3's open weights are 594 GB in BF16 and need about 1.68 TB of VRAM to serve; the API lists at $3/$15 per 1M tokens. Here is the 2026 self-host vs API break-even math with real GPU rates.
AMD's ROCm 7, plus SCALE and HIPIFY, make some CUDA workloads portable to AMD GPUs. But custom kernels, TensorRT-LLM and NCCL still bind enterprises to NVIDIA. What actually moves off CUDA in 2026.
Kimi K3's API costs $3/$15 per million tokens. Self-hosting the 2.8T open weights needs roughly 1.4 TB of VRAM. We size the GPUs, price the node against 2026 cloud rates, and work out the break-even.
vLLM, Ollama and LM Studio solve different problems. Red Hat measured vLLM at 793 tokens/s versus Ollama's 41, and local inference costs about $0.18-0.29 per million tokens. Throughput, GPU cost, and when each wins.
vLLM's April 2026 benchmark put FP8 KV-cache 14.9% above BF16 on one H100. Priced out, that is $4.24 down to $3.69 per million output tokens - and your AWS purchase model moves the number four times further.
