Infrastructure
63 articles RSS
Google Announces XLA Serving Engine, a C++ Runtime for Exported Models; Its Benchmark Reports 2x to 5x a Python Server's Throughput on Small Requests
Google's blog launches XLA Serving Engine, a C++ runtime for exported models. Its repo benchmark, run on one TPU chip, lists 2.0x to 5.0x a Python server's throughput for 1- and 16-item requests and 1.14x at 128.
Google Open-Sources TPU Sync, a C++ KV-Cache Transfer Library for TPUs; Its Own Low-Concurrency Tests Report Average TTFT Cuts of 60% and 90%
Google says TPU Sync, an open-source C++ library for KV-cache transfer on TPUs, cut average TTFT 60% and 90% at low concurrency in internal July 2026 tests; its README says the APIs are evolving rapidly.
PyTorch 2.15 Makes a Rewritten NCCL Backend the Default; Release Notes Say Groups Create Communicators Eagerly and Allocate GPU Memory Even if Unused
PyTorch 2.15.0 (October 28, 2026) makes a rewritten NCCL backend the default; its notes list eager communicators that allocate GPU memory and say nccl-legacy will be removed in the next release.
Google and Meta Open-Source TorchTPU, a PyTorch Backend for TPUs Using a tpu Device, a tpu_dist Backend and Three Eager Modes
InfoQ reports that TorchTPU, built by Google with Meta, lets PyTorch run on Google TPUs; its repository lists Python 3.11+ and release 0.2.0 on PyPI.
What Inference Speedup Claims Disclose: MLPerf Rounds, Project Benchmarks and MLCommons' Planned Shift to Served-Endpoint Testing
Independent coverage of MLPerf Inference v6.0 and v6.1 notes limits on comparing results, MLCommons says Endpoints will replace its datacenter benchmark, and project figures from vLLM and Valkey are self-reported.
Watermarks, Routing, Harnesses and Sampling Settings: Which Controls Recent AI Platform Moves Leave With Developers and Which With Providers
Reports on watermarks, model routing, a managed agent harness and removed sampling parameters show some settings fixed by the provider and others left to developers.
TRL 1.17.0 Adds Mistral Large 4 Training on a Single 8xB300 Node and Makes Score Centering the Default AsyncGRPO Objective
Hugging Face's TRL v1.17.0 adds a single-node LoRA example for Mistral Large 4, changes the AsyncGRPO default objective to score centering and makes GRPOConfig raise on top_p or min_p without top_k when vLLM importance sampling is on.
NVIDIA Dynamo 1.6.0 Adds Pluggable Router Flow Control and LPX Kubernetes Components, and Deprecates the Router's Scoring Flags
Dynamo v1.6.0, published October 22, adds request-classifier plugins to its router, LPU-backed LPX components for Kubernetes, and snapshot-backed engine failover, and moves to SGLang v0.5.21 and vLLM v0.31.0.
llm-d 1.0.0 Graduates Its Router Configuration API to v1, Aligns Component Versions and Deprecates Its ROCm Image
llm-d v1.0.0, published October 20, aligns component versions across its organization, moves the router configuration API to v1, adds DisaggregatedSet and deprecates its own ROCm image build.
OpenAI Adds a 12-Hour Prompt Caching Option to the Responses API, With Cache Writes Priced at 2.5 Times the Input Rate
OpenAI's API changelog says developers can now choose 12-hour prompt caching; its guide lists a 2.5x cache-write rate against 1.25x for the 30-minute default.
Valkey Project Benchmarks a Shared Valkey KV Cache for LLM Inference, Reporting 25% More Requests Served on the Same Two GPU Nodes
A Valkey blog post reports that a three-shard Valkey 9 cache served 4,631 requests versus 3,692 on the same two 8-GPU nodes, at a cost of about 15% on top of the GPU bill.
DeepSpeed 0.19.8 Removes Elastic Training, Progressive Layer Dropping and Apex AMP, Turning Leftover Configs Into Errors
DeepSpeed 0.19.8 removes elastic training, Progressive Layer Dropping, Apex AMP and graph harvesting; configs that still set them now raise DeepSpeedConfigError.