Disaster Recovery and Microservices: The BAC Theorem
Microservices follow the polyglot persistence principle, where every microservice manages its own persistence independently. In this talk we illustrate the ultimate consequences of these assumptions, which can be summarized using the BAC theorem: only two are possible out of 1) a backed up microservice architecture; 2) full availability during normal operations; and 3) consistency after recovery. In other words, we will show that only Microservices Architecture running without a Backup can be both Available while remaining Consistent after disaster strikes. We will present and compare several coping strategies to deal with this limitation and discuss how it affects the monolith decomposition process at design time and the operational coupling between different microservices at run time.
Abstractions
Microservices
Will this component
always terminate?
function f() {
...
return 42;
}
Development
Will this service
run forever?
while (true) {
on f {
return f()
};
}
Operations
Will this microservice continuously change?
while (true) {
on f {
- return f()
+ //return f()
+ return f2()
};
}
DevOps
Microservices
The microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own container and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies.
Martin Fowler and James Lewis
Microservices
The microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own container and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies.
Martin Fowler and James Lewis
For us service orientation means encapsulating the data with the business logic that operates on the data, with the only access through a published service interface. No direct database access is allowed from outside the service, and there’s no data sharing among the services.
One team has full control of the entire devops (code, build, test, release, deploy and operate) cycle
Iterate fast: Many small frequent releases better than few large releases
Loosely Coupled Microservices
Avoid dependencies: If you have to hold a release until some other team is ready you do not have two separate microservices
Avoid cascading failures: A failed microservice should not bring down the whole system
Do you:
Operate more than one microservice?Use polyglot persistence?Avoid storing everything in the same database?Assume eventual consistency?
Microservices
Microservices prefer letting each service manage its own database, either different instances of the same database technology, or entirely different database systems - an approach called Polyglot Persistence.
Microservice architectures are doomed to become inconsistent after disaster strikes
Devops meets Disaster Recovery
How do you back up a monolith?
How do you back up one microservice?
How do you back up an entire microservice architecture?
Are you sure?
Example
Data relationships across microservices = Hypermedia
Independent Backup
Backups taken independently at different times
Disaster Strikes
Disaster Strikes
One microservice is lost
Recovery from Backup
Broken link after recovery
Eventual Inconsistency
Synchronized Backups
Backups of all microservices taken at the same time.
Limited Availability
No updates allowed anywhere while backing up the microservices
The BAC theorem
When Backing up a microservice architecture, it is not possible to have both Consistency and Availability
Consistency
During normal operations, each microservice will eventually reach a consistent state
Referential integrity: links across microservice boundaries are guaranteed not to be broken
Availability
It is possible to both read and update the state of any microservice at any time
Backup
While backing up the system, is it possible to take a consistent snapshot of all microservices without affecting their availability?
No.
Backup + Availability
Backing up each microservice independently will eventually lead to inconsistency after recovering from backups taken at different times
Backup + Consistency
Taking a consistent backup requires to:
disallow updates anywhere during the backup (limited availability)
wait for the slowest microservice to complete the backup
agree among all microservices on when to perform the backup (limited autonomy)
Shared Database
A centralized, shared database would require only one backup
Is this still a microservice architecture?
Shared Database, Split Schema
A centralized, shared database would require only one backup
Each microservice must use a logically separate schema
What happened to polyglot persistence?
Links can break
No guarantees for references crossing microservice boundaries
Microservices inherit a fundamental property of the Web
Orphan State
Orphan state is no longer referenced after recovery
Unstoppable System
An expensive, replicated database with high-availability for every microservice
Unstoppable System
How do you restart an unstoppable system?
Eventual Consistency
Retries are enough to deal with temporary failures of read operations, eventually the missing data will be found
Eventual Inconsistency
Retries are useless to deal with permanent failures of read operations, which used to work just fine before disaster recovery
Distributed Transactions
Take snapshots only when all microservices are consistent
Avoid eventual consistency
Microservices
Distributed transactions are notoriously difficult to implement and as a consequence microservice architectures emphasize transactionless coordination between services, with explicit recognition that consistency may only be eventual consistency and problems are dealt with by compensating operations.
Ana Ivanchikj, Cesare Pautasso, and Silvia Schreier, Visual modeling of RESTful conversations with RESTalk, Journal of Software & Systems Modeling, pp. 1-21, May, 2016. (Journal-First Best Paper Award at MODELS 2016)
Jim Webber, Savas Parastatidis, Ian Robinson, REST in Practice: Hypermedia and Systems Architecture, O'Reilly, 2010
Cesare Pautasso, Erik Wilde, Push-Enabling RESTful Business Processes, 9th International Conference on Service Oriented Computing (ICSOC 2011), Paphos, Cyprus, December 2011
Cesare Pautasso, BPMN for REST, In Proceedings of the 3rd International Workshop on BPMN (BPMN 2011), Lucerne, Switzerland, November 2011
Guy Pardon, Olaf Zimmermann, Florian Haupt, Silvia Schreier, Ana Ivanchikj, Mathias Weske, Adriatik Nikaj, Sankalita Mandal, Hagen Overdick, Jesus Bellido, Rosa Alarcón, Alessio Gambi, Daniele Bonetta, Achille Peternier, Erik Wilde, Mike Amundsen, Stefan Tilkov, James Lewis