Cloudflare Expands Workers AI to Host Trillion-Parameter Models With New Infire Engine and Unweight Compression
Cloudflare detailed the custom infrastructure behind its Workers AI expansion to host models like Kimi K2.5, including a Rust-based inference engine and a lossless weight-compression system.
Overview
Cloudflare has detailed the engineering infrastructure it built to run extra-large open-source language models on its global edge network. In a series of blog posts published in March and April 2026, the company described two new technologies — Infire, a custom Rust-based inference engine, and Unweight, a lossless model-weight compression system — along with architectural changes that enabled it to bring Moonshot AI’s Kimi K2.5, a model with over 1 trillion parameters and roughly 560 GB of weights, into production on Workers AI.
What We Know
A Custom Inference Engine Built for the Edge
Cloudflare began building Infire in 2025 because existing open-source inference runtimes did not fit its deployment model. According to the company’s engineering blog, the core constraint was architectural: “There’s a mismatch between Cloudflare’s globally-distributed network and a typical centralized AI deployment using large multi-GPU nodes.” Standard solutions such as vLLM also lacked a feature Cloudflare required: “vLLM does not support co-hosting multiple models on the same GPU without using Multi-Instance GPU (MIG), and we need to be able to dynamically schedule multiple models on the same GPU to minimize downtime.”
Infire is written in Rust and uses continuous batching, paged KV cache, and just-in-time kernel compilation. On H100 NVL hardware using the ShareGPT dataset at 200 concurrent requests, Infire achieved 40.91 requests per second at 25% CPU load, compared to vLLM 0.10.0’s 38.38 requests per second at 140% CPU load, according to Cloudflare’s August 2025 benchmark post. The company summarized the engine’s advantage as: “Infire completes inference tasks up to 7% faster than vLLM 0.10.0 on unloaded machines,” with GPU utilization reaching “upward of 80%, reducing our operational costs.”
For large-scale deployments, Cloudflare extended Infire with pipeline-parallel and tensor-parallel modes alongside expert parallelism. The result is that Infire can run Kimi K2.5 on 8 H100 GPUs with more than 30 GiB still available for KV-cache, and can load the largest models in under 20 seconds.
Disaggregated Prefill and Decode
Beyond the inference engine itself, Cloudflare introduced a prefill-decode (PD) disaggregation architecture for Kimi K2.5. The approach separates the two computation phases onto different machines: as Cloudflare explained, “Prefill is usually compute bound, while decode is memory bound,” making them poor fits for the same hardware. InfoQ’s coverage of the announcement noted that this enables independent tuning and scaling of each stage.
The performance impact was significant. After switching to the disaggregated architecture, p90 time per token went from ~100 ms with high variance to 20-30 ms, a 3x improvement in intertoken latency, and this improvement occurred while request volume increased using the same quantity of GPUs.
Prompt caching also improved alongside the disaggregation work. Cloudflare introduced a session affinity header (x-session-affinity) to route repeat requests to inference nodes that already hold computed input tensors, which raised input token cache hit ratios from 60% to 80% during peak times.
Kimi K2.5 on Workers AI
Cloudflare announced the availability of Moonshot AI’s Kimi K2.5 on Workers AI in March 2026. The model carries a 256k context window with support for multi-turn tool calling, vision inputs, and structured outputs. Cloudflare described internal testing results in which a security review agent processing over 7 billion tokens daily cut costs compared to mid-tier proprietary models: “Running this agent with Kimi K2.5 cost just a fraction of that: we cut costs by 77%.”
Cloudflare’s efficiency improvements also extended to smaller models. According to InfoQ, Llama 4 Scout now runs on just two H200 GPUs on the platform.
Unweight: Lossless Compression Without Accuracy Loss
Published the day after the performance post, Cloudflare’s April 17, 2026 blog described Unweight, a compression system targeting GPU memory bandwidth constraints. “Running LLMs across Cloudflare’s network requires us to be smarter and more efficient about GPU memory bandwidth. That’s why we developed Unweight, a lossless inference-time compression system that achieves up to a 22% model footprint reduction.”
The system exploits a statistical property of BF16 floating-point weights. Each BF16 value contains a sign bit, an 8-bit exponent, and a 7-bit mantissa. In practice, “the top 16 exponents account for 99% of all model weights,” meaning “you only need ~2.6 bits to represent this distribution — far less than the 8 bits allocated.” Unweight applies Huffman coding to just the exponent byte: “We leave the sign and mantissa untouched and compress only the exponent byte using Huffman coding — a classic technique that assigns short codes to common values and longer codes to rare ones.”
Applied to Llama 3.1 8B, Unweight delivers “~30% compression of Multi-Layer Perceptron (MLP) weights alone,” translating to “15-22% in model size reduction and ~3 GB VRAM savings.” The distinction between the two figures matters operationally: the system achieves “~13% model footprint reduction for inference bundles” versus “~22% for distribution bundles” due to differences in how weights are packed for distribution versus serving.
The compression is not free. The company acknowledged that “Unweight is not a free lunch. On-chip reconstruction adds computational work that wouldn’t exist with uncompressed weights” and that “the inference configuration saves approximately 13% of total model memory at a throughput cost of roughly 30% at typical serving batch sizes.” To mitigate this, the system runs decompression inside GPU on-chip shared memory rather than the higher-bandwidth HBM, and an autotuner selects the optimal decompression pipeline per weight matrix and batch size.
Unweight is currently integrated with Cloudflare’s Infire engine and targets MLP weight matrices, which represent roughly two-thirds of model parameters. The company has identified further optimization opportunities, noting it has not yet compressed the down projection in each MLP layer — about one-third of the compressible weights.
What We Don’t Know
Cloudflare has not disclosed which additional open-source models it plans to add to Workers AI or a timeline for resolving Unweight’s remaining 30–40% throughput overhead. The company acknowledged that “new technologies, research, and models come out on a weekly basis for the machine learning community” and that it is “continuously optimizing” its stack, but no roadmap was published alongside the announcements. It is also not clear whether Unweight’s GPU kernels will be released as a standalone open-source project separate from the research paper already published.
Analysis
The combination of Infire and Unweight reflects a broader pattern among infrastructure providers seeking to differentiate on efficiency rather than raw GPU count. By compressing models in a lossless way and disaggregating prefill from decode, Cloudflare is attempting to serve larger models without proportionally larger hardware budgets. The 3x latency improvement delivered through PD disaggregation using the same GPU quantity is the most operationally significant claim, as it means performance gains without additional capital expenditure.
The tradeoff in Unweight’s current form — 13% memory savings at 30% throughput cost — limits deployment to scenarios where memory pressure outweighs throughput demand. Cloudflare’s own framing acknowledges this is an early-stage system with known optimization gaps. The broader significance may be in the research direction: using information-theoretic analysis of weight distributions to find compression without rounding or approximation is a different approach from the lossy quantization methods more commonly used in production inference today.