Kafka Streams

From Producers to Real-Time Topologies When you first step into the world of Apache Kafka, it feels like walking into a busy airport. Producers are like airplanes constantly landing with new passengers (data), Consumers are like buses waiting to pick up those passengers, and Kafka Streams is the control tower that orchestrates real-time decisions. In this blog, I’ll walk you through Kafka’s journey — starting from basic Consumers, touching on scalability and offsets, and then moving into the Kafka Streams API,…

Read MoreKafka Streams

The Rise of Kafka and Stream Processing – moving data

As a System Designer, let me take you through the journey of how data processing eventually gave birth to Kafka and real-time stream processing. In the early days, data processing was basic and primitive: capture the data, store it somewhere, and then process it later. I think big data is not the invention but a large data set of certain domain being accumulated over a period of time. When big data came along, the pattern didn’t really change. We just started…

Read MoreThe Rise of Kafka and Stream Processing – moving data

Notification Service

A notification system is responsible for sending timely alerts to users through different channels .A notification system has become one of the most essential features in modern applications, enabling instant communication between platforms and their users. Whether it is a bank sending a payment confirmation, an e-commerce app updating on delivery status, or a news portal sharing breaking updates, notifications ensure timely information reaches the right users across multiple channels like email, SMS, push, or in-app alerts. Beyond just information delivery,…

Read MoreNotification Service

High Availability

Picture this: you open your favorite streaming app on a Friday night, popcorn in hand, all set to binge that new series everyone’s talking about. You hit play—and nothing happens. The app just spins. After a few refresh attempts, you give up and head to Twitter to vent about it. That frustration you just imagined? That’s what happens when a system isn’t “highly available.” High availability (HA) might sound like a fancy buzzword, but at its core, it simply means: your…

Read MoreHigh Availability

URL Shortner

URL shortening is a technique on the World Wide Web in which a Uniform Resource Locator (URL) may be made substantially shorter and still direct to the required page. This is achieved by using a redirect which links to the web page that has a long URL [wikipedia] We’ve all seen them. Those neat little links that look like this: 👉 Instead of a giant wall of text like: 👉 https://www.superlongwebsite.com/products/category/page?id=928374&ref=campaign&utm_source=ads&utm_medium=email&utm_campaign=something-2025 Cleaner, right? Easier to share, tweet, and even remember. That’s the magic of URL shorteners. But here’s the…

Read MoreURL Shortner

Key Value Store

Key value store the first thing flashes in mind is “Map” data structure which are actually designed to store key value pairs. Key-value stores might sound simple, but they’re the quiet workhorses powering some of the biggest systems on the planet. —the simplicity hides enormous power. Key-value stores are the backbone of many large-scale systems we use every day, from caching in Facebook’s feed to Amazon’s shopping cart. As a system designer, I absolutely love talking about key-value stores because they’re…

Read MoreKey Value Store

Consistent Hashing

consistent hashing is the distributed hashing technique that distributes data across multiple servers in the cluster. It is used to efficiently rebalancing of the distribution of data across cluster when nodes are added ore removed (minimize data movement). It maps data and nodes on to hash ring. NoSQL database like Cassandra is the best example of use of consistent hashing. When we have fixed size cluster of nodes, below simple hashing works great, serverIndex = hash(key) % N, where N is…

Read MoreConsistent Hashing

Rate Limiting

Rate limiter is the controller to monitor and control the amount of traffic allowed to protect the network or your network of the services from attacks and overwhelming the system and eventually avoid it from crashing down. In the context of HTTP, it restricts the number of requests a client can send within a specified time period. Once the threshold is exceeded, any additional requests are blocked. What is Rate Limiting, Really? Restricting or putting certain controls over how callers can…

Read MoreRate Limiting