System Design Basics - Master Message Queues in Just 5 Minutes!
"Need to grok message queues? This 5-minute guide has you covered!
Hello guys, Message Queues like IBM MQ, Apache Kafka is one of the most essential System Design concept for any developer who wants to design asynchronous and loosely coupled system.
If you are new here, In the past, I have talked about common system design concepts and components like Rate Limiter, Database Scaling, API Gateway vs Load Balancer and Horizontal vs Vertical Scaling, Forward proxy vs reverse proxy, and Single Point Failure, and in this article we will talk about Message queues which are very important for many Event Driven architecture.
For this article, I have teamed up with Hayk and we'll dive into the fundamental concepts of Message Queue, and provide practical insights to help you make informed decisions about your Software Architecture and System Design.
By the way, if you are preparing for System design interviews and want to learn System Design in depth then you can also checkout sites like ByteByteGo, Design Guru, Exponent, Educative, Codemia.io, and Udemy which have many great System design courses
Now, let’s start with the Message Queues and I am sure in 5 minutes you will have good idea of what they are and why you need them?
Imagine an online store where every time a customer places an order, you need to:
Process the payment.
Update inventory.
Send a confirmation email.
Doing all of this immediately, especially during peak traffic times, could slow down the customer’s experience.
In this case, we have a large number of application events, and we can’t handle all of them at once.
Of course, we could scale up our servers to handle these large amounts of application events, but if we don’t have to handle all of them at once, it’s better to queue these events and handle them later.
Basic Architecture of a Message Queue
A message queue is a durable component stored in memory that supports asynchronous communication. It serves as a buffer and distributes asynchronous requests.
The basic architecture of a message queue is simple. Input services, called producers or publishers, create and publish messages to a message queue.
Other services, called consumers or subscribers, connect to the queue and perform actions defined by the messages.
In a real-world scenario, there can be many apps writing to the queue and many servers reading from the queue.
Back to the example
So, in our case, instead of handling each task immediately, you can add it to the back of the queue, and from this queue, they are sent to our servers.
Order placed: The order details are put into a message.
Message sent: The message is added to the queue.
Workers process: Separate processes (workers) pull messages from the front of the queue and handle the tasks.
Also, our server acknowledges that it received and processed one message, and the queue removes it so that it is not sent for the second time.
Benefits of using Message Queues
The main advantage is that we decouple these events, and this message queue will allow us to process these events asynchronously. We can queue them until we can process them.
With the message queue, the producer can post a message to the queue when the consumer is unavailable to process it.
Also, the consumer can read messages from the queue even when the producer is unavailable.
Another great benefit is that they are durable. If the queue crashes, that data will not be lost as it’s not stored in RAM but in Disk.
If a worker crashes while processing a message, no problem! The message is still in the queue and will be picked up by another worker.
Message queues also provide scalability. If you receive a flood of orders, the queue will just get longer. You can add more workers to handle the extra load without affecting the website.
Different Queue Types
There are multiple types of message queues. The most common ones are:
FIFO (First-In-First-Out): Just like a regular line, messages are processed in the order they arrive. This is important for things like payment processing.
Priority Queues: Some messages might be more important than others. You can prioritize these so they get processed sooner.
Push vs Pull
Some queues wait for workers to ask for messages (pull-based queue), while others actively send messages to workers (push-based queue).
Examples
Here are some popular examples of message queues:
RabbitMQ: A versatile queue that’s good for many use cases.
Kafka: Built for high throughput and real-time data streaming. Great for things like logging and event-driven architectures.
Amazon SQS (Simple Queue Service): A fully managed cloud-based queue service offered by AWS. It’s scalable and reliable, with features like delay queues and dead-letter queues.
Here are more differences between Kafka, ActiveMQ and RabbitMQ, three of popular Message Queues available to us
If you’d like to see how message queues can be used in action when designing large-scale distributed systems, then be sure to check out this post next, where we design a system of a Twitter-like app.
Want to level up your system design skills with Hayk ‘s personal guidance? Join his community and gain access to weekly lessons, real-world examples, and hands-on guidance.
Other System design articles you may like to read
 | A guest post by
|