Google Announces XLA Serving Engine, a C++ Runtime for Exported Models; Its Benchmark Reports 2x to 5x a Python Server's Throughput on Small Requests

Overview

Google announced the launch of XLA Serving Engine (XSE) in a Google Open Source Blog post dated Thursday, November 5, 2026 (the page shows no time zone). The post describes it as “a general-purpose model export and inference runtime for the OpenXLA ecosystem, open-sourced under the Apache 2.0 license”. Models are exported from Python and then served by a C++ runtime, a workflow the post calls “Train in Python, Serve in C++”. The Register, an outlet independent of Google, covered the launch in an article published at 11:30 UTC on November 11, 2026.

The headline performance figures are Google’s own. The post says that for short requests XSE delivers “2–5x the throughput of the best Python server we could build”. The benchmark page in the project repository lists 2.0x for 16 items per request and 5.0x for 1 item per request (the Herald’s reading of the page’s table; the post gives only the range), and 1.14x at 128 items per request, all measured on a single TPU chip.

What XSE is

According to the project’s README, XSE is “a C++ inference runtime and model packaging toolkit for the OpenXLA ecosystem”. A model is exported once from JAX (Flax, Keras 3 or plain jax.jit) or from PyTorch via torchax, and the resulting bundle is served on CPU, NVIDIA GPU or Cloud TPU “with no Python in the request path”. The Google post says the exported “XLA SavedModel” holds the model’s computation as StableHLO, with weights stored as an Orbax checkpoint. The project’s overview page says the runtime is a C++ binary built on OpenXLA’s PJRT and IFRT device interfaces. The README says prebuilt server images are published for CPU, CUDA 12, CUDA 13 and Cloud TPU, and the Google post says they expose gRPC APIs.

The scope is deliberately narrower than that of LLM servers. Google’s post says XSE serves models “typically non-LLMs”, calls it “a lightweight, one-shot inference engine” and recommends vLLM or SGLang for large autoregressive language models with long context lengths. The README lists recommenders and rankers, vision and multimodal encoders, encoder-style language models and short fixed-horizon decoding as its target workloads. The Register says XSE “is not so much suited for large-scale LLM inference”, and quotes Nathan Beach, a Google director of product management, as saying that a significant amount of production inference is still ranking, recommendation, embedding and vision models.

Google’s post also makes these statements about use and plans:

• XSE serves “over 6 million queries per second” and powers pipelines including YouTube Shorts, Google Ads, Search and parts of Gemini Omni.
• Google plans to integrate with TorchTPU so that PyTorch users can export models to an XLA SavedModel for C++ serving on TPUs. The Herald covered TorchTPU on November 4, 2026.
• Ahead-of-time compilation is being rolled out to Google’s internal users before external ones, and a C++ data preprocessor is on offer for users who want to combine data processing with the model’s graph.

The Register also places XSE next to existing options, noting that PyTorch offers a native C++ runtime called LibTorch and that Microsoft and its collaborators offer ONNX Runtime. The project’s own overview page argues that converting a model to ONNX, TensorRT or TensorFlow via jax2tf often breaks, or needs a separate inference model definition per hardware backend, when the model uses custom XLA calls, Pallas kernels, multi-device sharding annotations or mixed-precision numerics.

Google’s benchmark, as published

The repository’s benchmark page serves one recommendation model, DLRM-DCNv2, on a single Cloud TPU v6e-1 VM, with every server pinned to the same 8 host CPU cores. It reports throughput in examples per second while keeping 99% of requests under 20 ms, as medians of three 30-second runs with zero errors. The benchmark code is published in the repository’s examples/dlrm_benchmark directory.

Items per request	XSE (examples/s)	Best Python server (examples/s)	XSE vs. best Python	Single-process Python (examples/s)
1	30,100	6,100	5.0x	845
16	83,400	42,400	2.0x	10,000
128	92,400	81,000	1.14x	53,100

On memory, the benchmark page reports about 4x less RAM for XSE than for the 8-process Python server (8.0 to 8.4 GiB against 31.9 GiB), because each Python process holds its own copy of the 3.2 GB embedding tables. The same page says a custom Python deployment could avoid that duplication through copy-on-write fork() or hugepage-backed /dev/shm mappings. The page’s summary states 30% to 70% less host CPU time per prediction than that server, while Google’s post ties that range to short requests. The page’s per-request-size figures are 0.224, 0.059 and 0.041 CPU-seconds per 1,000 examples for XSE against 0.770, 0.084 and 0.057 for the multi-process Python server; the Herald’s own arithmetic on those figures gives about 71%, 30% and 28% less, the last for the 128-item row, which falls just below the stated range.

The benchmark page lists its own limits. All Python setups ran on CPython 3.12 with the global interpreter lock, and free-threaded builds and the experimental JIT “were not evaluated”. The Python setups used grpcio, NumPy and jax.jit without custom C++, Cython or Numba extensions. Results come from one hardware slice, and sparse IDs were drawn uniformly at random. With a single client, the single-process Python server answered 1-item requests in 1.23 ms (median) against 1.18 ms for XSE with batching off. The page also notes that for pre-batched 128-item requests, the multi-process Python server reaches 88% of XSE’s throughput under the 20 ms budget, and it advises turning off server-side batching in that case.

What is not known

• Independent measurements. None of the cited sources reports a throughput measurement of XSE made by anyone other than Google. The Register repeats Google’s figures, introducing them as “Google’s tests”. The post’s partner statements from Arm, Intel and Moloco, who are described as private-preview partners, carry no comparative figures.
• Other models and hardware. The benchmark page reports one model on one hardware slice (a single TPU v6e chip). It includes reproduction instructions for GPU and CPU, but the page shows no GPU results.
• When the code became public. The post is dated November 5, 2026, and The Register, published November 11, gives XSE the sub-headline “Newly open sourced”. The repository’s release pages show v0.0.12 dated 2026-10-06 22:12:37 UTC and v0.0.13 dated 2026-11-03 17:54:26 UTC, both before the post. None of the cited sources says when the code first became public, and the Herald has not established it.
• Date of Beach’s post. The Register says Beach wrote on LinkedIn on “Tuesday”; the Herald did not read that post.

For how other inference speedup claims disclose their test set-ups, see the Herald’s earlier Analysis, What Inference Speedup Claims Disclose.