Kafka Streams

From Producers to Real-Time Topologies When you first step into the world of Apache Kafka, it feels like walking into a busy airport. Producers are like airplanes constantly landing with new passengers (data), Consumers are like buses waiting to pick up those passengers, and Kafka Streams is the control tower that orchestrates real-time decisions. In this blog, I’ll walk you through Kafka’s journey — starting from basic Consumers, touching on scalability and offsets, and then moving into the Kafka Streams API,…

Read MoreKafka Streams

Kafka Architecture

Kafka Internals: How Topics, Partitions, and Replicas Actually Work When you start building real-time streaming systems, you’ll inevitably hear “Kafka is the backbone of modern data architectures.” But what does that really mean under the hood? Let’s break it down step by step, so you not only know what Kafka does, but also why it was designed this way. First Things First: What Is Apache Kafka? At its simplest, Kafka is a distributed message broker. It sits in the middle of…

Read MoreKafka Architecture

The Journey from Big Data to Real-Time Streams

If you’re building modern systems, chances are you’ve already heard the buzzwords: Big Data, Hadoop, Kafka, Spark, Streaming. But have you ever wondered where this whole movement started and why stream processing even became a thing? The Pre-Google Era: When the Web Was Indexed by Hand Back in the early ’90s, the World Wide Web wasn’t searchable the way we know it today. In fact, until 1993, Sir Tim Berners-Lee maintained a hand-curated list of websites at CERN. No crawlers, no…

Read MoreThe Journey from Big Data to Real-Time Streams

The Rise of Kafka and Stream Processing – moving data

As a System Designer, let me take you through the journey of how data processing eventually gave birth to Kafka and real-time stream processing. In the early days, data processing was basic and primitive: capture the data, store it somewhere, and then process it later. I think big data is not the invention but a large data set of certain domain being accumulated over a period of time. When big data came along, the pattern didn’t really change. We just started…

Read MoreThe Rise of Kafka and Stream Processing – moving data