Read the full article here.
Databases have been traditionally segmented into analytical and transactional capabilities. We’re increasingly seeing the market move toward the middle.
Venn diagrams have always struck me as an easy, accessible way to observe distinct features and overlap between two entities. As an experienced database engineer, I’ve found Venn diagrams to be useful tools when illustrating the database market as well.
On one side of the market, we have analytical databases. These are designed to handle business intelligence capabilities and are good at “bulk loading” and storing large amounts of data. An analytical database is a great tool for telling you about your sales trajectory over time or your inventory levels.
On the other side, we have transactional databases. These solutions are best for ensuring strong data integrity and handling high throughput, low latency writes. Meaning these systems can process lots of small changes to a data set quickly and with little friction. For example, when a bank wants to associate all transactions—ATM withdrawals, checking deposits and more—to a single customer, a transactional database is able to organize these features.
Over the past 10 years, we have increasingly seen the overlap between these two sides of the market expand, forming a new middle segment. Real-time databases are, by their very definition, at the intersection of transactions and analytics. They require certain aspects of both types of workloads to succeed. When creating real-time analytics databases, database engineers essentially drew a vertical line and asked, "What are the core capabilities needed to succeed at both of these jobs?" The answer? A lot, including JSON querying, low latency streaming data ingestion and flexible indexing, to name a few.
It has been gratifying to see that the broader market is validating our thesis: The middle lane is the one to drive in.
The market share of real-time analytics databases continues to expand. Additionally, Snowflake, Google and MongoDB are all trying to develop combined analytical and transactional platforms. Even the divide between NoSQL and SQL applications is being bridged. Yet, these solutions may not be the long-term leaders in converged databases, as they are mostly bolting support onto existing mature architectures instead of building to support both from the beginning as a core tenant of their design. They are merging toward the middle lane instead of driving in the center lane from the start. Supporting both analytics and transactions makes it harder to take "design shortcuts" to optimize for one or the other.
The race to the middle is on at full speed, and not everyone will be able to win. It’s also true that not every application may need a database designed for transactions and analytics. And that is okay. But databases emerging in the middle of these historically divided halves are increasingly gaining prominence and are a segment of the market to remain attuned to going forward.
The potential benefits of more general-purpose databases are many, and you should look for these capabilities when searching for or building a database system for your organization. General-purpose databases enable capabilities in an application that a specialized database wouldn't allow such as exposing analytical results in an application. This is hard to do if an app is built using a transactional database only because it's too slow.
Another great example is adding generative AI capabilities to an application that uses vector search, full-text search and SQL querying, all of which operational databases don't have support for.
Finally, a more general-purpose database can simplify building applications because you don't need to build a data pipeline on moving data between various specialized databases in order to build some capabilities.
One thing is for sure: This is an exciting time to be in the database industry. I know I can't wait to see how it will all pan out.
If you like this article consider subscribing to our bi-monthly newsletter to get information about our portfolio, solutions, and insights delivered to your inbox.