Database CEO on why moving to a “State Transition Database” is like taking the red pill in the matrix

In today’s episode of the Brains Byte Back podcast, we are joined by Dave Remy, CEO of Event Store DB, an open-source state-transition database, designed for businesses that are ready to harness the power of event-driven architecture.

You will learn why Remy advocates that we need a new category of database now and what makes this type of database different. Alongside, how you can leverage the event store platforms to drive growth, efficiency, and innovation. And finally, some practical ideas and pitfalls of a data engineer when implementing event storage databases.

You can listen to the episode below, or on Spotify, Anchor, Apple Podcasts, Breaker, Google Podcasts, Stitcher, Overcast, Listen Notes, PodBean, and Radio Public.

Alternatively, you can find a transcript below:

Dave: Thank you for having me on. My name is Dave Remy, I’m the CEO of Event Store. I’ve been in the industry dating myself for 25 or 30 years now, I would say, depending on how you count, both at tech companies like Microsoft, IBM, and BA Systems, primarily on the dev tool side, but also in startups, a couple of startups as well as being an enterprise software as a CIO and engineering manager. Probably more interesting to the list of listeners is that is Event Store itself. event store is the company behind the popular open source database technology, Event Store DB, as well as which we’re going to talk more about in this podcast. So we will talk more about that later. And then as well as Event Store Cloud, which is the managed service that for companies that don’t want to run Events Store up themselves, they can come to the team that is responsible for event store to run Event Store DB.

Sam: Fantastic. Well, thank you for coming on the show today. You clearly have a long history in this line of work. And I’m really curious to know like when and how did Event Store DB first start?

Dave: Yeah, so Event Store DB has actually been around for a while it was open sourced in 2012. So it’s been around for over 10 years now. It was originally created by Greg Young, who is one of the discoverers, who made popular, this idea of event sourcing, and a team of other pioneers that built the first database, you know, pioneers in the event sourcing community that built this first database, they, they had created this this type of database repeatedly for other projects at the time, and they decided it was time to create a production ready version of it, kind of distilling what they’ve learned from prior projects. The first customer and sponsor of the project was this pharmaceutical company based in Bath UK called bath ASU, which was a spin out Bath University. But since then, 1000s of companies around the world have adopted Event Store DB as a production, operational database technology, pretty much across every industry stage of company. So it’s come a long way since that beginning in 2012.

Sam: That’s awesome. And that’s a complete curveball, because Bath that’s city or Bath, as like many people call it as well, in where it’s from, I say it with a bit of an accent, really. But it’s actually very close to where I’m from Bristol, where you can actually cycle Yeah, they’re so close to each other. Yeah, that’s crazy. That’s awesome. So also, I’d be really interested to know like, can you share some of the latest advancements in event sourcing technology? And how Event Store like fits into the larger picture?

Dave: Yes, absolutely. So events sourcing , as I mentioned, 2012 for the database, and it’s been around before that. So it’s it’s continuing to mature over the years and gain adoption on this stuff that the pattern itself has mature. And as more and more companies have adopted it, we’re seeing Best Practices improve and success rates improve. I would say one area that has been improving Is the analysis techniques that have emerged, things like event storming and event modeling, which are these new ways that business users can participate in and visualize the design of, of the systems in a way that’s very understandable by business users. I mean, it turns out that designing a system around events is a powerful and a natural way to design a system that the as I mentioned, the business users can understand and participate in. One of the unique aspects of event sourcing is that the these business understandable events that are in a sequence, they’re they’re very similar to what the way business people think about their business. And what you surface in the design phase ends up showing up all the way down into the actual database itself. It’s a it’s fairly unique versus traditional databases. In technical jargon, I say the design artifacts are isomorphic with the physical artifacts, which just means that you’re actually taking the exact thing business users are dealing with and putting them in the database, which makes, which has the benefit of developers speaking the same language that business users do. And so we’ve seen that, that that maturing of how you design these types of things and engagement with business users. Event Store DB itself was built from the ground up for event sourcing this technique will talk a bit about more than I’m sure further on. But one of one of the profound aspects of event sourcing is that it proposes a new, a new, more fundamental way of storing data of keeping data on keeping which is keeping all the changes in in the data, not just keeping the current state which we’ll talk about a bit more. It turns out this is highly relevant to the major trends that are going on in the software industry around microservices and in data distributed data, which we’ll talk about a bit more furthermore

Sam: Awesome. Sounds like you’re already like pushing boundaries there. And I’m curious to know like how does Event Store DB fits into that overall, like event driven architecture?

Dave: That’s a great question. And Event Store DB is an event oriented database, obviously, you can tell that by the name of it, even though we haven’t talked a lot, a lot about it. And it can be foundational in an overall event driven architecture. And those of your listeners that know about event driven architecture, they know how important and impactful of a concept, it’s it has been, and is influencing really companies across the board right now. But however, as it mostly as it’s implemented today, it’s defined sort of outside the application boundary outside the core systems. It’s between systems. It’s primarily an integration or a data extraction view of the world, with the application itself being this black box that happens to be emitting events. But but that view if you implement just that part, this leads to really difficult and even intractable problems around things like synchronisation and audit and lineage of data. And once it goes outside the applications. For example, if you think about it, if a key decision is made, because an event is fired in an application, it can be impossible within traditional database to figure out what happened because it’s already forgotten all of its history. So with Event Store DB, there’s dirt, it’s within the application. And it’s an opportunity to have events, not just at the edge of applications, but all the way to the foundation of an application. And imagine that you’re actually writing and your core database within the application is event oriented. And now it becomes foundational for an entire event driven architecture for the for the whole enterprise. And that’s what we’re seeing more and more is this dissatisfaction with just putting event driven architecture at the edge of applications and more adopting it at the core of an application. For the reasons that I mentioned earlier?

Sam: Okay, cool. Cool. Now, I have to admit, like, I have somewhat of a shallow knowledge across all areas of technology having hosted this podcast, and we touch on a number of topics. So I will admit, like, when it comes to this stuff, I’m not the most tech savvy, but I would be curious to know, like, why do we need a new category of database now? And what makes this type of database different?

Dave: Yes. So let’s start with a part of why what makes this database different. So it’s pretty straightforward to understand the traditional databases that are out there for operational data, the relational database, you’ve heard of the graph database, MongoDB, with the document database, they are key focus on keeping current state, the state of the world as it is. So you know, if you update that data, it throws away the the old update that was updated and puts in the new data. Or if you delete data, of course, that goes away. Now, Event Store DB is it’s a State Transition Database, which may sound complicated, but it’s really pretty straightforward. It keeps you keep all of the changes in state, not just the current state, and you keep them in order, and including the sequence number that’s associated with each of those changes. Now, if you have all the changes in state, you can always kind of plate replay that up to current and see the world as current databases would. But the these changes are first class and they’re kept in the in the database in a way that traditional tradition databases don’t. The other thing there is that the business name is kept as well. So these changes in state. They’re called events because they’re named. So for example, in a traditional database, or say, you have the last name changed from Remy to Smith, in a traditional database, you might just capture that you throw away, Smith was there and only Remy would be there. But this could be this would be captured in the database for the Event Store DB. And it would be given a name like say, for example, marital status changed event, which could be very different from the last name that’s changed. That could be called say a name corrected event, two different things not captured in any current database technology that I know of, and yet critical for downstream systems that are that maybe need to reason about this data. Maybe your AI doesn’t care at all about corrective events. They just want to know what marital status changed events. These types of scenarios, and many more start to be possible with an Event Store DB and technology like this. The other thing that is significant is that traditional operating databases were meant to pull data with Event Store DB. The database is built to stream data it streaming is first class. So if you want to with a traditional database, if you wanted the data if you want the data, you need to go to the database, and you need to query it and pull it out when you want it. But Event Store DB like lives in a distributed data world where data is constantly streaming to multiple downstream and data syncs data data technologies in your enterprise, and allows you to do that in real time and stream that data. So when you’re trying to move to a real time enterprise, like so many businesses are, this becomes a critically valuable capability. And finally, I think kind of the coup de gras here is that, because you’ve kept all of the changes, in addition to being able to stream them in real time, you can replay them from any point in time, including the beginning. So I mean, the implications of that are profound if you think about it from the flexibility, business flexibility that you get, because you stored all of these changes. And now when business requirements change, you want a new report, you want to change from one analytics technology to another, you can replay those changes from the beginning, and repopulate that and have those reports as if they’d always been there from the beginning of time. So it’s a it’s really amazing for optionality and business flexibility and agility.

Sam: Yeah, I think I’m getting a better understanding of this now. And I really liked that example you gave with the changing of the surnames that that made things a lot clearer actually. Can you also go into detail for our listeners on how they can leverage the event store platforms to drive growth, efficiency and innovation for their companies?

Dave: Yes, certainly. That’s, you know, event sourcing and Event Store DB have been adopted in some of the fastest growing and innovative companies in the world right now, including Walmart, zero.com, National Cash Register. Many of the top payment providers, some of which I can’t speak about, because we’re under confidentiality, but many more. And I would say, despite them being cross industry and across every shape and size, what they have in common is that they they see the importance and the value of this alternative, the this idea of, of your storing data, and what you know about it, and if we’ll use an analogy that comes up quite often is this from the Matrix movie, they they’ve taken the red pill, once they become aware that this option exists, and that they could keep the data on, you sort of can’t unsee it, and it starts to seem negligent using a traditional database, because you’re throwing data away, or what I tongue in cheek call intentional data loss architecture. And so what adopting a platform like Event Store DB becomes foundational for driving, driving growth, efficiency and innovation and the business environment we’re in that is constantly changing, and needs to be able to respond to AI and other trends that are going on in the industry.

Sam: Fantastic. Well, I have to say, I love that matrix analogy. And I’m pretty sure most of our listeners will as well. One of my last questions you really I want to know like what are some like practical ideas and pitfalls of a data engineer when implementing event and storage databases?

Dave: Yeah. So from a practical perspective, I would say is to pray, I think that anybody who’s in the software industry or on the business side, that depends on data, I would encourage you to proactively go out and learn about event sourcing and Event Store DB, if you try it out. Education and training is available through event store, but also through many other channels. A point I would like to make for someone who’s thinking about this adopting this is that oftentimes, when you first hear about this, you think, oh, especially if you buy into it, if you take the red pill, you think, okay, I need to adopt this wholesale, that I’m that I’m we’re proposing that, you need to go out and take your entire Oracle database or your SQL Server database, and transfer it over to Event Store DB wholesale, that’s not required at all and not recommended. And Event Store DB is built for connection. So which means you can start small, you can you can build your first name application, and then because it streams stream back to any other users or that data, which might even include the legacy system that you’re trying to replace a part of, for example. So start small, you know, adopt it for a small project, both personally and for your company. And then only then, you know, take it and grow and learn incrementally in a safe way. And that’s entirely possible with this technology. And then another thing I would say is, of course, this is data and data is important. It’s important asset for your business. So and you’re you’re adopting this sort of foundational data model of keeping low level data. So the best practices of data engineering still apply. And design matters a lot because the better your initial design, the better you’ll be able to leverage the data in ways you might not imagine when the system was created. So that I mentioned this before, but a State Transition Database like Event Store DB, it preserves optionality, it allows you to move quickly in the face of business change, which we all know is an inescapable fact of life. And one final pitfall that I’ll mention it may not be, it may not be that critical for this audience. But we see a lot of times you explain this. And it seems simple to have this type of data model. And we see, we see often, companies and customers attempt first to build this themselves, they go out and think, Oh, I’m just going to store all the changes, I can do that I can put it in my file system, or I can kind of crowbar it into some other database technology. But as you might imagine, first of all, a database is required. And if there’s a lot of specific challenges to this type of thing, it’s really a piece of infrastructure. And so when you’re ready, you really do want a production ready database, not something that you’ve built bespoke, which which could run, you could run a lot of risk when you try to put production data into it.

Sam: I think you said some really solid advice there. And I hope our listeners can take something from that. And really, my last question to you is, I want to know, like, what are you folks focusing on now what’s next on the horizon for Event Store dB?

Dave: So as I mentioned Event Store DB is in production now in 1000s, of projects around the world, both in Event Store Cloud, where we hosted but also hosted directly by companies of all shapes and sizes. Most of our current focus is on helping Event Store DB and event sourcing, in general to be easier to adopt across every major programming language. At the moment, Event Store DB has supported clients for pretty much every major language, TypeScript, JavaScript, Java, C, sharp, go and Haskell, Python is very soon. And it’s already in a beta type of mode, and other languages are coming. So it should be available in pretty much any language that you’re interested in. And then another area is to continue to focus on database tooling to help event store work with downstream technologies. So at the moment, you have, this requires a lot of custom development, when you want to connect up to different technologies, different downstream data technologies, what we’ll be coming out, and what we envision is sort of this out of the box connection capability to be able to transform data and connect in real time to all of the major downstream data technologies. And of course, we continue to push on scalability and stability of the core database, we’ve been running an event store cloud for well over a year. And that just running it ourselves is enabled us to find more ways to improve stability and scalability that will benefit all Event Store DB adopters, whether it’s in cloud or whether it’s not in cloud. So all in all, the future looks like really excited for events, the events are dv, we believe it’s going to impact the industry and in fundamental, profound and fundamental ways and positive ways. So we hope folks will come in, give it a try.

Sam: Yeah, I would highly encourage anyone listening, that is interested to go give it a try. And on that note, like if people are interested, how can they get in contact with you and keep up with the work you folks are doing?

Dave: There’s two major ways, the best way is just go to Eventstore.com. And there all of the contact methods are in there. It is an open source technology, you can go to the GitHub site and see all of what’s happening, what types of issues are occur occur, and also pointers to forums if you want to join a forum and and talk to the community more directly. So that’s all available via eventstore.com.

Sam: Fantastic. Well, we’ll include a link to that in the show notes. But otherwise, I want to say Dave, thank you so much for joining me today. And yeah, I wish you everyone there the best of luck.

Dave: Thanks so much. Appreciate it.

Disclosure: This episode includes a client of an Espacio portfolio company