Cloudflare opens a public beta for K2 event streams

Cloudflare has opened a public beta of K2, a durable event-streaming service that sits between producers and consumers and holds on to their events until somebody reads them. The company announced it on 1 October, describing K2 as a streaming primitive for its developer platform rather than another message broker.

The problem K2 is meant to solve

Cloudflare's starting point is the mismatch built into ordinary remote procedure calls. A producer and a consumer have to agree on scale and on time. If events arrive faster than the consumer can process them, or the consumer is simply down, they are dropped. That gets worse the moment more than one system needs the same events, as when an ecommerce backend emits a transaction event that an analytics service and a fraud detector both need. The fix is to decouple the two sides by putting something in the middle that absorbs writes and lets readers work at their own pace.

A log on top of object storage

K2 is that something. Events are written to a stream, which keeps them as an ordered log; consumers can split reads across a group or take every message themselves, and long retention means a consumer can be offline for a long stretch without losing data. Underneath, K2 is a partitioned durable log built on R2 object storage, which is how Cloudflare gets the volume it needs out of a system that never drops an accepted event.

Building a log on object storage means solving an obvious problem: R2, like other object stores, cannot append. Writes are accumulated in memory on an edge service and flushed as segment files large enough to justify the cost of writing and reading them, with ordering and strictly incrementing offsets handled through R2's atomic operations rather than a separate coordination service. The trade-off is produce latency. Cloudflare puts it at about one second at the 99th percentile in this first release, and says a technical deep dive is coming.

Why not run Kafka at the edge

The design comes from Cloudflare's own need rather than a market gap. K2 began life as the durable buffer for Basin Pipelines, which processes events on a pull model and therefore needs somewhere to store them first. The obvious answer would have been Apache Kafka, except that Pipelines runs on Cloudflare's edge, spread across more than 335 cities, where machines are small, relatively ephemeral and often talk to each other over the public internet. Rather than fight that, Cloudflare pushed replication and consensus down into R2 and kept the application layer simple, which also separates compute from storage so each can scale on its own.

K2, Queues or Pipelines?

Cloudflare already sells two adjacent products, and the post spends time drawing lines between them. Queues is built around individual units of expensive work, with retries, delays and dead-letter queues for failures, which suits something like an image-processing job. K2 is built for high-scale data movement with long retention. Basin Pipelines remains the stream-processing layer that reads, transforms and writes results to R2.

Our opinion

The interesting thing here is not that Cloudflare shipped a queue with longer memory. It is that the company keeps bending its own infrastructure into shapes the rest of the industry runs on dedicated hardware, and then selling the result as a primitive anyone can call. Rebuilding a log's ordering guarantees on atomic object-storage operations is a genuinely unusual bet, and the one-second p99 produce latency is the price tag sitting on it. That is a tolerable number for analytics and fraud detection and an impossible one for anything interactive, so K2's public beta should be judged on whether the latency comes down as the design matures. If it does, the pattern is a tempting one: durable storage is the hard part of distributed systems, and Cloudflare is betting it can borrow somebody else's.