Apache Software Foundation Graduates Rust Streaming Platform Iggy and AI-Provenance Tool Sourcelume to Top-Level Status
The ASF elevated Apache Iggy, a Rust-based message streaming platform, and Apache Sourcelume, an AI training-data provenance tool, out of its Incubator on August 27.
Overview
The Apache Software Foundation announced on August 27 that two projects, Apache Iggy and Apache Sourcelume, have graduated from its Incubator to become Top-Level Projects (TLP), according to the ASF. The announcement, dated from Wilmington, Delaware, covers a Rust-based message streaming platform built for low-latency workloads and a new instrumentation layer aimed at documenting the origins of AI training data.
What We Know
Apache Iggy is described by the foundation as “a hyper-efficient, persistent message streaming platform written in Rust, built for predictable ultra-low latency and a minimal infrastructure footprint,” according to the ASF. The project is built around a thread-per-core, shared-nothing architecture paired with the Linux io_uring interface, an approach the foundation says eliminates userspace locks on the hot path, garbage-collection pauses, and unnecessary thread contention. Iggy natively supports the TCP, QUIC, HTTP, and WebSockets protocols, along with SDKs across multiple programming languages and an extensible connector ecosystem, according to the ASF.
Kranti Parisa, Apache Iggy’s PMC Chair, said in the announcement that “Iggy can process millions of messages per second and hundreds of terabytes of data per day on a single node, while maintaining single-digit millisecond tail latency,” adding that its architecture “continues to be validated through demanding real-world workloads and continuously published benchmarks,” according to the ASF. Parisa said reaching Top-Level Project status “gives Iggy a strong, independent, community-governed foundation for the future.”
Apache Sourcelume takes a different focus: it provides “open source instrumentation for AI training-data provenance, giving dataset producers, model builders, and downstream consumers a shared, vendor-neutral way to document where training data came from and under what terms it may be used,” according to the ASF. Rather than creating a new metadata format, Sourcelume adds an attestation and registry layer on top of existing dataset-description standards, allowing the metadata those standards already produce to be signed, published, and independently verified, the foundation said.
Jamie Goodyear, Apache Sourcelume’s PMC Chair, said “training data is the foundation everything else in AI is built on, and right now that foundation is largely undocumented,” according to the ASF. Goodyear added: “A signed record isn’t an endorsement — it’s accountability. A producer can state — in a standard, cryptographically signed way — exactly what they’re asserting, and anyone downstream can verify who said it and that it hasn’t changed.”
Both projects passed through the Apache Incubator, which the foundation describes as the entry path into the ASF for incoming projects, known as podlings, providing them with services and mentorship to build durable communities and adopt what the foundation calls “The Apache Way.” A podling graduates to Top-Level Project status once it has demonstrated “an active, diverse contributor base and a track record of independent, consensus-driven governance,” according to the ASF.
The Apache Software Foundation, established in 1999, describes itself as the global home for open source software, and its portfolio includes projects such as Apache Airflow, Apache Camel, Apache Cassandra, Apache Groovy, Apache HTTP Server, and Apache Kafka, according to the foundation.
What We Don’t Know
The announcement does not disclose how many organizations currently run Iggy or Sourcelume in production, nor does it name specific companies or products adopting either project. The ASF’s announcement also does not specify how long each project spent in incubation before graduating.
Analysis
The two graduations land on opposite ends of the current infrastructure conversation around AI. Iggy addresses the plumbing problem of moving large volumes of data quickly and cheaply, a need the foundation frames explicitly around “real-time, data-intensive, and AI workloads.” Sourcelume addresses a governance problem that has grown alongside AI’s data appetite: knowing where a model’s training data actually came from. Both becoming Top-Level Projects in the same announcement gives each a standalone, foundation-backed governance structure independent of the corporate sponsors that originally incubated them, a status the ASF ties to a project having built a self-sustaining contributor community rather than remaining dependent on a single backer.