Event bus for dummies

As a software developer at Heureka Group, I'm offering a brief introduction to software architecture, recipes, and insights from my experience. I highly recommend this article to all developers and product managers. It's a valuable resource for anyone looking to enhance their understanding of software architecture.

What is the primary purpose of software architecture?

I believe that creating a highly organized application structure is the primary objective of all software engineers. We want to develop sustainable, maintainable, and expandable solutions. As developers, we should always keep the future in mind.

Many applications during development become something like a plate of spaghetti. There are a lot of connections between vast amounts of services. This is not precisely an architecture – it is only a plate of spaghetti. No one knows what is connected where. No one has a complete picture of the application.

The only solution to untangle the spaghetti structure is to find an Italian meal with a better structure. We are looking for something structured and layered – what about lasagna?

The Story of SW Architecture

A monolithic solution is a prime example of a spaghetti structure. Such a monolith structure has long grown, regardless of how developers, technologies, and architectural philosophy have changed during its development. 

Microservices – a way out of spaghetti

Microservices are the only way out from the monolithic spaghetti knot. So, we have separated the code and its data into simple, independent applications. 

The threat: Rest Nest

Once you break a monolith into services, it works nicely. It will separate the functionality into logical parts, standardize communication between services (REST), and everything is fine.

But the new idea brings new problems. The usual one is when one service calls another service, and it calls other services, etc. This case creates undefined response times and complicates monitoring of what is happening in the system at any given moment. Also, past states of the system are lost forever. 

 This is something like the higher level of spaghetti architecture (I personally call it Rest-Nest).

When an architect is in a tight corner, he always uses one solution (except suicide, of course). He adds another layer. So, let's use our microservices structure and glue them using another layer, and we finally get a nice portion of lasagna. 

Request-driven vs. event-driven

The microservices with REST interfaces are typical Request-driven architecture. In this type of architecture, users or services send commands to the system. The command tells the system what to do.

On the other hand, Event architecture describes what happens in the system. The message describes the change as an event. The state of the system is described by a chain of events. Let's call it the event bus.

The event bus is a system built around the bus. There are applications that send data to the bus, called producers, and applications that read data from the bus, called consumers.

Event bus design patterns

Design patterns are bricks we use for building complex systems.

• Producer= The producer sends event messages to the bus.
  
  

• Consumer = The consumer reads event messages from the bus.

• Dead letter queue

When an incoming message can't be processed for any reason, the system cannot stop processing queues or simply forget about it. So, we always create a special event to store these "invalid" messages.
Typical case: Message is not JSON valid

• Processor

When an application consumes a message and then produces another message, we call it a processor.
Typical case: We have a user ID and want to add more data about the user to the message.

• Materializer

The application collects incoming message data to the database. We call this app materialiser.

Typical case: Visit counter: If our event bus contains one message for one user visit on the web, we can count it using materialiser.

• Retry queue

When there is some third-party service that is not reliable (in extreme, this case is all services), we need to store the message for the following process round.
Typical case: The invoice system generates invoices, which is time-consuming. So, the processor needs to implement a retry queue to wait for the generated invoice.

Pros of event bus architecture

• Small apps
  The system is divided into applications as small as possible.

• Extensibility
  Adding (or removing) this new app to this structure is easy – it is extensible. Great for the future, remember?

• Scalability
  We can have (nearly) an unlimited amount of consumer app instances.

• Dependency
  Apps do not depend on each other. Apps depend only on incoming events – easy-to-test

Cons of event bus architecture

• Big infrastructure
  Even a tiny solution needs significant infrastructure.

• Self-description
  The event bus requires precise specification that describes Event flow.
  The structure and meaning of events are not a part of the event bus.

• Centralized
  When the bus is down, everything is down.

• Latency
  There could be a long way from the start to the end of event processing.

• Async
  It is always asynchronous. It could be mostly an advantage and sometimes a pain – it is hard to mix with synchronous systems.

Experience

Our project contains 15+ services and 30+ event messages. It is not such a simple task to, e. g., change the event name throughout the whole project. So these are conventions we decided to follow.

Conventions

• One topic per event
  Every topic contains only one type of event.
  Why: Events have different importance, count, and configuration (e. g., retention in Apache Kafka)

• One publisher per topic
  There could be only one event producer.
  Why: solves questions: Who sent this?

• Consumer as simple as possible
  The consumer has to be as simple as possible. Reads data, processes data, sends or resends data, nothing else.

• Processor as fast as possible
  Processors could be the bottlenecks. It slows the flow of events.
  Why: The processor is in the middle of two events. So, its processing time is a delay in event flow. Don't forget to measure processing time.

Event naming

Event names are made from two parts – the subject and the action, written in dot notation.

subject.action

The subject is typically a singular noun – the name of processed content (such as credit, payment, invoice, notification) or service name (for dead-letter-queue). 

The action is a verb in past tense and describes what happened with the subject (such as created, requested, expired, confirmed, canceled).

For example:

• card.expired
• credit.changed
• credit.spent
• directPay.responded
• freeCredit.charged
• invoice.created
• invoice.failed 

Documentation

Every service specification or subsystem contains an Event flow chart made in Mermaid, allowing us to join these small charts into the big one.

The complete bus event flow chart also works as the interface between developers and project managers. It easily allows you to check different use cases.

Summary

Now that you understand the benefits of using an event bus and you had a look at how to implement it, do not hesitate. And please hop on the bus.

Sources:

• The Concept of Event-Driven Microservice Architecture by Sumith Fernando
• Event-Driven Architecture: Getting Started with Kafka by Jean-François SIMON