Building Event-driven Microservices Using CQRS and Serverless

This blog series will introduce you to building event-driven microservices as cloud-native applications.

In this first post, we’ll explore how to implement the CQRS pattern in microservices. We’ll also dive into why serverless is a natural fit for these kinds of systems. Later in the series we’ll explore a reference application that uses Spring Cloud Stream to implement CQRS.

What is an event-driven architecture?

Event-driven architectures treat domain events as first-class citizens. This approach is as old as software itself.

One example we use every day is in front-end applications. In every web browser in use today, events are handled as a way to capture inputs of a user form. Events connected to page elements are handled by an explicitly mapped function, sometimes referred to as an action or command, which will apply state changes to a user interface when triggered.

Now, more recently, with the widespread adoption of microservices, there is renewed interest in how to take advantage of event-driven techniques in distributed back-end systems.

CQRS

One of the most popular practices in event-driven architectures today is called CQRS, which is short for Command Query Responsibility Segregation. CQRS is a style of architecture that allows you to use different models to update and read domain data.

The basic idea of CQRS is that it’s perfectly natural to need to separate the models you’re using to update and read data. The diagram above shows this basic idea.

Building Spring Cloud Microservices That Strangle Legacy Systems

It’s safe to say that any company who was writing software ten years ago—and is building microservices today—will need to integrate with legacy systems. In this article, we will explore building cloud-native microservices that still need to integrate with legacy systems. We’ll use practices from Martin Fowler’s Strangler Application to slowly strangle domain data away from a legacy system using microservices.

When building microservices, the general approach is to take existing monoliths and to decompose their components into new microservices. The most critical concerns in this method have much less to do with the application code and more to do with handling data. This article will focus on various methods of strangling a monolith’s ownership of domain data by transitioning the system of record over time.

Throughout this article, we’ll use a reference application built with Spring Boot and Spring Cloud. The example demonstrates techniques for integrating a cloud-native microservice architecture with legacy applications in an existing SOA.

Going Cloud Native

Many companies want to start taking advantage of the public cloud without having to migrate every line of business application at the same time. The reasons for this are numerous. The existing line of business applications can be thought of as the vital organs of a living organism. During the migration, think about the complex relationships between the existing components deployed to your infrastructure. Think about the dependencies of your applications and the connections between them. Think about every application that relies on a database or network file system. These are among the many considerations that will cause a migration to become an expensive and time-consuming project.

The unfathomable complexity of migrating applications to the cloud can delay a decision from being made until the business deems necessary, which is usually triggered by a major event that results in a loss of revenue. This ticking time bomb will result in a lift-and-shift migration of applications. The problem with the lift and shift approach is that any technical debt that you had on-premises finds new life in the cloud environment. The problem being, the architectural and infrastructure issues that triggered the cloud migration still may not be fixed.

The approach I discuss in this article focuses on addressing the underlying symptoms of legacy systems that decrease system resiliency and lead to costly failures.

Event Sourcing in Microservices Using Spring Cloud and Reactor

When building applications in a microservice architecture, managing state becomes a distributed systems problem. Instead of being able to manage state as transactions inside the boundaries of a single monolithic application, a microservice must be able to manage consistency using transactions that are distributed across a network of many different applications and databases.

In this article we will explore the problems of data consistency and high availability in microservices. We will start by taking a look at some of the important concepts and themes behind handling data consistency in distributed systems.

Throughout this article we will use a reference application of an online store that is built with microservices using Spring Boot and Spring Cloud. We’ll then look at how to use reactive streams with Project Reactor to implement event sourcing in a microservice architecture. Finally, we’ll use Docker and Maven to build, run, and orchestrate the multi-container reference application.

Eventual Consistency

When building microservices, we are forced to start reasoning about state in an architecture where data is eventually consistent. This is because each microservice exclusively exposes resources from a database that it owns. Further, each of these databases would be configured for high availability, with different consistency guarantees for each type of database.

Eventual consistency is a model that is used to describe some operations on data in a distributed system—where state is replicated and stored across multiple nodes of a network. Typically, eventual consistency is talked about when running a database in high availability mode, where replicas are maintained by coordinating writes between multiple nodes of a database cluster. The challenge of the database cluster is that writes must be coordinated to all replicas in the exact order that they were received. When this happens, each replica is considered to be eventually consistent—that the state of all replicas are guaranteed to converge towards a consistent state at some point in the future.

When first building microservices, eventual consistency is a frequent point of contention between developers, DBAs, and architects. The head scratching starts to occur more frequently when the architecture design discussions begin to turn to the topic of data and handling state in a distributed system. The head scratching usually boils down to one question.

How can we guarantee high availability while also guaranteeing data consistency?

To answer this question we need to understand how to best handle transactions in a distributed system. It just so happens that most distributed databases have this problem nailed down with a healthy helping of science.

Transaction Logs

Mostly all databases today support some form of high availability clustering. Most database products will provide a list of easy to understand guarantees about a system’s consistency model. A first step to achieving safety guarantees for stronger consistency models is to maintain an ordered log of database transactions. This approach is pretty simple in theory. A transaction log is an ordered record of all updates that were transacted by the database. When transactions are replayed in the exact order they were recorded, an exact replica of a database can be generated.

The diagram above represents three databases in a cluster that are replicating data using a shared transaction log. The zipper labeled Primary is the authority in this case and has the most current view of the database. The difference between the zippers represent the consistency of each replica, and as the transactions are replayed, each replica converges to a consistent state with the Primary. The basic idea here is that with eventual consistency, all zippers will eventually be zipped all the way up.