inference

Google's blog launches XLA Serving Engine, a C++ runtime for exported models. Its repo benchmark, run on one TPU chip, lists 2.0x to 5.0x a Python server's throughput for 1- and 16-item requests and 1.14x at 128.

Google says TPU Sync, an open-source C++ library for KV-cache transfer on TPUs, cut average TTFT 60% and 90% at low concurrency in internal July 2026 tests; its README says the APIs are evolving rapidly.

Independent coverage of MLPerf Inference v6.0 and v6.1 notes limits on comparing results, MLCommons says Endpoints will replace its datacenter benchmark, and project figures from vLLM and Valkey are self-reported.

Dynamo v1.6.0, published October 22, adds request-classifier plugins to its router, LPU-backed LPX components for Kubernetes, and snapshot-backed engine failover, and moves to SGLang v0.5.21 and vLLM v0.31.0.

OpenAI and Anthropic list one model at several per-token rates by speed, OpenAI prices GPT-6.1 Sol at a fifth of Astra, and Red Hat reportedly caps developer token spend.

ONNX Runtime 1.31.0 promotes model-package and EPContext data APIs to stable, deprecates the ACL execution provider, and makes out-of-range GatherBlockQuantized indices return zeros.

SGLang v0.5.21 lets PD instances switch between prefill and decode without a restart, makes a Rust prefix-cache core the default, and removes deprecated endpoints.

Nvidia's 64GB DGX Spark ships October 23 from partners from $4,999, with unchanged bandwidth and clustering, while the 128GB model's price rose to $6,950 amid a memory shortage.

GMI Cloud says it raised $223 million in Series B equity led by ARCHIV plus a $445 million CTBC-led credit facility to expand GPU and inference capacity in the US, Taiwan and APAC.

Crusoe's oversubscribed round, co-led by Atreides, Mubadala and Valor, funds modular Spark data centers and its Managed Inference platform, which claims to outpace vLLM.

AMD and Embedded LLM tested five speculative-decoding drafting methods in vLLM on AMD Instinct GPUs, with some configurations reaching up to 2.87x throughput over the non-speculative baseline.

NVIDIA's Groq-derived Groq 3 LPX inference accelerator is now shipping, promising ultrafast token generation for agentic coding workloads, with Nebius first to deploy it.