Weaviate 1.38 Promotes Its Disk-Based HFresh Vector Index to GA, Cutting Memory Use for Billion-Scale Search

Overview

The open-source vector database Weaviate shipped version 1.38.0 in early June, promoting its disk-based HFresh vector index to general availability. According to the project’s GitHub release notes, HFresh is now marked “GA” and arrives with “numerous performance optimizations reducing memory usage, disk writes, and allocations.” The 1.38.0 release was published on June 5, as listed in a weekly catalog roundup by Elestio. The release notes record no breaking changes.

HFresh targets one of the most expensive parts of running a vector search system at scale: keeping the full set of high-dimensional vectors in memory. By moving those vectors to disk, the index aims to let teams index far larger datasets on the same hardware.

What We Know

HFresh first appeared as a technical preview in Weaviate 1.36, released on March 3, according to the Weaviate blog. The blog describes HFresh as “a new disk-based vector index inspired by the SPFresh algorithm.” Rather than holding every vector in RAM the way the default HNSW index does, HFresh “divides vectors into small regions called postings — groups of vectors that are close to each other in vector space, stored on disk in an LSM store,” the blog explains.

Searches run in two stages. As the Weaviate blog describes it, “a compact in-memory HNSW index over the centroids identifies which regions” of the vector space are relevant to a query, and then “only the corresponding postings are fetched from disk and searched in detail.” Because “HFresh only needs the centroid index and metadata in memory — the full vectors live on disk,” the project says the design yields “significantly lower memory usage compared to HNSW, with I/O bounded and latency predictable even as the dataset grows into the billions.”

Weaviate’s documentation describes HFresh as “a cluster-based vector index based on the SPFresh algorithm” that “uses an HNSW index for centroid search, providing a balance between memory efficiency and search performance.” The docs list it as the fourth available vector index type alongside HNSW, the flat index intended for low object counts, and the experimental dynamic index that starts flat and converts to HNSW once a threshold is crossed.

The SPFresh algorithm that HFresh draws on was published in the academic paper “SPFresh: Incremental In-Place Update for Billion-Scale Vector Search”. Its central claim, per the paper’s abstract, is that it “provides superior query latency and accuracy to solutions based on global rebuild, with only 1% of DRAM and less than 10% cores needed at the peak compared to the state-of-the-art, in a billion scale vector index with 1% of daily vector update rate.”

Beyond HFresh, the 1.38.0 release adds several capabilities still labeled as previews in the release notes. Namespaces “add control-plane and data isolation between users on a shared cluster.” Nested object filtering “adds ability to perform search within indexed JSON properties.” A schema reindexing feature “adds support for changing property’s index types.” The release also introduces a “Secure Built-In Model Context Protocol (MCP) server interface,” which the notes say “allows AI agents (such as Claude, IDEs, etc.) to natively read and write to Weaviate without requiring custom code.”

What We Don’t Know

The release notes promote HFresh to general availability but, in the text reviewed for this article, do not quantify the memory reduction in 1.38 relative to earlier previews. Weaviate’s documentation frames HFresh’s benefit qualitatively as “a balance between memory efficiency and search performance” rather than with a specific figure. The namespaces, nested object filtering, schema reindexing, and MCP server features remain previews, and the release notes caution generally that preview features may change. Independent benchmarks comparing HFresh’s GA performance against HNSW on production workloads were not available at publication time.