Enabling new class of data-intensive apps

A new class of apps are pushing the boundaries of technology these days, commonly known as data-intensive apps. A data intensive app is where the primary challenge is quantity, complexity and speed of data and its changes -- as opposed to compute-intensive apps where CPU cycles are the bottleneck.

A reference architecture of this is below.

This new architecture while addressing some the market needs of data hungry apps creates a new set of challenges for the app developer around data consistency, manageability and operability of the apps. In response to this trend a new design pattern called microservices have arrived. This has been popularized by Netflix, Amazon and others. Microservices are still service-oriented architectures (SOA) composed of loosely coupled elements with bounded contexts. One best practise that has emerged in the design of microservices is the creation of separate datastore for each microservice.

Some characteristics of this new architecture is --

• Each microservice keeps an optimized view of the data locally.

• Each microservice can choose an optimal datastore (SQL DB, noSQL DB, Cache or Index) for their access pattern.

As a consequence, all the data stored in local stores have to become consistent in real-time. Hence recently, there has been a growing interest in change data capture (CDC), which is the process of observing all data changes written to a database and extracting them in a form in which they can be replicated to other systems. CDC becomes especially interesting if changes are made available as a stream, immediately as they are written. This is the exact problem that Replicant solves.

A cloud native, CDC like Replicant that can propagate data between different services without tight coupling (i.e. maintaining separation of concerns).