🧠 Humans of Cyber | Jay Kreps, Neha Narkhede, and Jun Rao
Distributed event streaming platform that stores and processes real-time data as ordered logs, enabling scalable messaging, analytics, and event-driven systems.
Apache Kafka was created at LinkedIn between 2008 and 2010 by Jay Kreps, Neha Narkhede, and Jun Rao to solve the limits of traditional messaging systems under massive user activity scale. It was open sourced in 2011 and became a Top-Level Project of the Apache Software Foundation in 2012, establishing vendor-neutral governance.
Kafka was designed to move organizations from batch processing to continuous event streaming. Rather than storing only the latest state of data, it persists ordered event logs, enabling replayability, auditing, and event-driven architectures.
At a technical level, Kafka organizes data into topics divided into partitions, each functioning as an append-only log. Producers write records containing key, value, timestamp, and headers, while consumers read using offsets, allowing independent and replayable processing. Replication across brokers provides fault tolerance, and durability is controlled through acknowledgment settings. Its core APIs include Producer, Consumer, Streams, and Connect for ingestion, processing, and integration.
Key milestones include Kafka 1.0 in 2017, Exactly-Once Semantics in 2018, the phased removal of ZooKeeper starting in 2021, full adoption of the KRaft metadata protocol in Kafka 4.0 in March 2025, and production-ready Share Groups in Kafka 4.2 in February 2026.
Kafka is deployed across cloud, hybrid, and on-prem environments and is widely used in finance, media, transportation, and retail for real-time analytics, fraud detection, telemetry, and microservices communication. It is licensed under the Apache License 2.0 and governed by the Apache Software Foundation through its Project Management Committee.
Subscribe and Comment.
Copyright © 2026 911Cyber. All Rights Reserved.
Follow 911Cyber on: