XX:YY

Your browser doesn't support the features required by impress.js, so you are presented with a simplified version of this presentation.

For the best experience please use the latest Chrome, Safari or Firefox browser.

image/svg+xml

Consistent Disaster Recovery for Microservices: the BAC Theorem

Cesare Pautasso
http://www.pautasso.info
c.pautasso@ieee.org
@pautasso

University of Lugano (USI)

Faculty of Informatics

Architecture, Design and Web Information Systems Engineering

http://design.inf.usi.ch

The BAC Theorem

Guy Pardon

Devops meets Disaster Recovery

image/svg+xml Backup Recover Code Build Test Release Plan Deploy Monitor Operate

Microservices

Microservices prefer letting each service manage its own database, either different instances of the same database technology, or entirely different database systems - an approach called Polyglot Persistence.

M. Fowler, J. Lewis https://www.martinfowler.com/articles/microservices.html

Eventual Inconsistency

Microservice architectures are doomed to become inconsistent after disaster strikes

How do you back up a monolith?

image/svg+xml Monolith Database Backup

How do you back up one microservice?

image/svg+xml microservice Database Backup

How do you back up
an entire microservice architecture?

image/svg+xml MySQL MongoDB Neo4J Redis

Are you sure?

Example

image/svg+xml Customer Product Shipment Order

Data relationships across microservices = Hypermedia

Independent Backup

image/svg+xml new C/1 C/1/name new C/2 C/2/name C/3/name new C/3 new O/3 O/3 → C/3 new O/1 O/1 → C/1 new O/2 O/2 → C/2 new O/3 O/3 → C/3 Customer Order new C/1 1 2 3 4 C/1/name new C/2 5 6 C/2/name new O/1 1 2 3 4 O/1 → C/1 new O/2 5 6 O/2 → C/2 1 2 3 4 5 6 1 2 3 4 5 6

Backups taken independently at different times

Disaster Strikes

Disaster Strikes

image/svg+xml new O/1 O/1 → C/1 new O/2 O/2 → C/2 new O/3 O/3 → C/3 new C/1 C/1/name new C/2 C/2/name Customer Order new C/1 1 2 3 4 C/1/name new C/2 5 6 C/3/name C/2/name new C/3 new O/1 1 2 3 4 O/1 → C/1 new O/2 5 6 new O/3 O/2 → C/2 O/3 → C/3 1 2 3 4 5 6 1 2 3 4 5 6

One microservice is lost

Recovery from Backup

image/svg+xml new O/1 O/1 → C/1 new O/2 O/2 → C/2 new O/3 O/3 → C/3 new C/1 C/1/name new C/2 C/2/name Customer Order 1 2 3 4 5 6 new O/1 1 2 3 4 O/1 → C/1 new O/2 5 6 new O/3 O/2 → C/2 O/3 → C/3 1 2 3 4 5 6 1 2 3 4 5 6 new C/1 C/1/name new C/2 C/2/name O/1 → C/1 O/2 → C/2

Broken link after recovery

Eventual Inconsistency

Synchronized Backups

image/svg+xml Customer Order new C/1 1 2 3 4 C/1/name new C/2 5 6 C/2/name new O/1 1 2 3 4 O/1 → C/1 new O/2 5 6 O/2 → C/2 new C/1 1 2 3 4 C/1/name new C/2 5 6 C/2/name new O/1 1 2 3 4 O/1 → C/1 new O/2 5 6 O/2 → C/2

Backups of all microservices taken at the same time.

Limited Availability

image/svg+xml Customer Order new C/1 1 2 3 4 C/1/name new C/2 5 6 C/2/name new O/1 1 2 3 4 O/1 → C/1 new O/2 5 6 O/2 → C/2 new C/1 1 2 3 4 C/1/name new C/2 5 6 C/2/name new O/1 1 2 3 4 O/1 → C/1 new O/2 5 6 O/2 → C/2

No updates allowed anywhere while backing up the microservices

The BAC theorem

When Backing up a microservice architecture,
it is not possible to have both
Consistency and Availability

Consistency

During normal operations, each microservice will eventually reach a consistent state

Referential integrity: links across microservice boundaries are guaranteed not to be broken

Availability

It is possible to both read and update the state of any microservice at any time

Backup

While backing up the system, is it possible to take a consistent snapshot of all microservices without affecting their availability?

No.

Backup + Availability

Backing up each microservice independently will eventually lead to inconsistency after recovering from backups taken at different times

Backup + Consistency

Taking a consistent backup requires to:

Shared Database

image/svg+xml Customer Product Shipment Order

A centralized, shared database would require only one backup

Is this still a microservice architecture?

Shared Database, Split Schema

image/svg+xml Customer Product Shipment Order C O S P

A centralized, shared database would require only one backup

Each microservice must use a logically separate schema

What happened to polyglot persistence?

Splitting the Monolith

image/svg+xml Customer Product Shipment Order

Keep data together for microservices that cannot tolerate eventual inconsistency

Unstoppable System

image/svg+xml Customer Product Shipment Order

An expensive, replicated database with high-availability for every microservice

Unstoppable System

How do you restart an unstoppable system?

Does it apply to you?

More than one stateful microservice

Polyglot persistence

Cross-microservice references

Disaster recovery based on backup/restore

Independent backups

Eventual inconsistency (after disaster recovery)

Does it apply to you?

More than one stateful microservice

Polyglot persistence

Cross-microservice references

Disaster recovery based on backup/restore

Synchronized backups (limited availability)

Consistent Disaster Recovery

Eventual Consistency

Retries are enough to deal with temporary failures of read operations, eventually the missing data will be found

Eventual Inconsistency

Retries are useless to deal with permanent failures of read operations, which used to work just fine before disaster recovery

The BAC Theorem

image/svg+xml Consistency Availability Backup CA CB AB Not Consistent Not Availablefor updates Not Backed Up

The BAC Theorem

When Backing up a whole microservice architecture, it is not possible to have both Consistency and Availability

Corollaries

  1. Microservice architectures eventually become inconsistent after disaster strikes when recovering from independent backups
  2. Achieving consistent backups can be attempted by limiting the full availability of the system and synchronizing the backups

References

Made with

http://asq.inf.usi.ch

Cloudware Engineering

IEEE Software - Special Issue

Call for Articles

Deadline: 1 April 2018

https://www.computer.org/software-magazine/2017/07/19/cloudware-engineering-call-for-papers/

Acknowledgements

Guy Pardon, Olaf Zimmermann, Frank Leymann, Florian Haupt, Silvia Schreier, Ana Ivanchikj, Mathias Weske, Adriatik Nikaj, Sankalita Mandal, Hagen Overdick, Jesus Bellido, Rosa Alarcón, Alessio Gambi, Daniele Bonetta, Achille Peternier, Erik Wilde, Mike Amundsen, Stefan Tilkov, James Lewis

Press space or use arrow keys to navigate

Powered by
asq