The Database Unbundled: Commit Logs in an Age of Microservices

GOTO Copenhagen 2019

When you examine the write path of nearly any kind of database, the first thing you find is a commit log: mutations enter the database, and they are stored as immutable events in a queue, only some hundreds of microseconds later to be organized into the various views that the data model demands. Those views can be quite handy–graphs, documents, triples, tables—but they are always derived interpretations of a stream of changes.
Zoom out to systems in the modern enterprise, and you find a suite of microservices, often built on top of a relational database, each reading from some centralized schema, only some thousands of microseconds later to be organized into various views that the application data model demands. Those views can be quite handy, but they are always derived interpretations of a centralized database.

Wait a minute. It seems like we are repeating ourselves.

Microservice architectures provide a robust challenge to the traditional centralized database we have come to understand. In this talk, we’ll explore the notion of unbundling that database, and putting a distributed commit log at the center of our information architecture. As events impinge on our system, we store them in a durable, immutable log (happily provided by Apache Kafka), allowing each microservice to create a derived view of the data according to the needs of its clients.
Event-based integration avoids the now-well-known problems of RPC and database-based service integration, and allow the information architecture of the future to take advantage of the growing functionality of stream processing systems like Kafka, allowing us to create systems that can more easily adapt to the changing needs of the enterprise and provide the real-time results we are increasingly being asked to provide.