Write up of Adrian Mouat’s talk about microservice deployment techniques from Devopsdays Edinburgh 2017.
Kubernetes is emerging as the platform within the industry for running containers. It is supported by Docker, DC/OS and CloudFoundry. Due to this rise in popularity higher level tooling is starting to appear but it is still early days.
Introduction
Before talking about the various deployment strategies available it is good to understand what we want from the ideal deployment strategy:
- Zero downtime - the obvious ideal
- Control over speed - we want to be able to get a deployment done as quickly or as slowly as we want, whether that is do everything now or do it gradually
- Control over traffic - we want to be able to control who can see a new version of an app and when, maybe we want to do internal first then some beta users then make it GA
- Observable - should be able to see a status of where the deployment is at and have metrics around how it is performing
- Easy to roll back - another obvious ideal, if somehow the new version doesn’t work we want to get back to the old working version quickly and easily
- Automatic - manually pressing buttons sucks especially when there is no need for it
- Fearless - if you have the above then you shouldn’t be scared about deploying new code, it becomes natural and not an event to fear
Once you know the ideal then it comes time to choose a strategy. Not every strategy fits every situation. The things to consider are:
- tolerance for error
- amount of testing
- rollout speed
- cost of deployments (not always £££, can be time etc)
Strategies
Kubernetes come with 2 default strategies for deployments:
- recreate - this is where you kill instances and then spin up new ones, similar to CloudFoundry, this causes downtime and is not ideal
- ramped deployment - no point where traffic isn’t being served, supports rollback, implies that you can handle 2 different codebases running at once, your apps need to shutdown cleanly, need to implement healthchecks otherwise app might not be ready for traffic but Kubernetes will be routing it regardless.
Going beyond the default options is possible but it will require more work on your part and it is all about what is best for you. Below are a couple of strategies that can be employed along with their pros and cons.
Blue/Green Deployment
Characteristics
- Spin up a new and entirely parallel version
- Test this new version
- Once happy flip entirely to new version
Pros
- Able to update multiple services together
- Test whole system before switching traffic
- Instant rollback
Cons
- Requires more resources
- Heavyweight
- No control over traffic or speed, all or nothing
- What happens with state? If your change includes database migrations how do you handle that?
- No obvious way to handle this natively in Kubernetes
Canaries
Characteristics
- Deploy new version to subset of users
- Monitor metrics
- Rollout wider when happy
Pros
- Avoids breaking changes for everyone
- Chance to monitor metrics
- Not as heavyweight as Blue/Green
Cons
- Requires intelligent load balancing
- More work to setup
- Slower rollout
It is possible to implement a canary like behaviour directly with Kubernetes commands however it is a little hacky and really should only be used in a pinch as it is very manual and you have limited control. The basic steps are:
- Start rollout
- Pause rollout
- Observe effect, monitor metrics
- Resume or undo rollout
Other
The following are not necessarily deployment strategies but they are useful to consider for their benefits.
PermaCanary
- Run multiple deployments with permanent canaries always running the next version
- HAProxy (or other load balancer) routing small percentage of traffic to the canary, say 5%
- This is discussed further in a blog post by Bitmovin.
A/B Testing
- Two or more versions running in parallel
- Metrics monitored to determine success - click rate, conversions etc
- Used for feature or design evaluation mostly
Feature Flags
- Multiple versions of execution path or design in codebase
- Code path selected dynamically
- Able to enable for certain users or a random sample
- Possible to do shadowing whereby multiple code paths are executed, only the old version is used to return to the user but allows new code to be verified
Conclusion
Every deployment plan will have concerns these typically include:
- API versioning
- Databases
- Deployment state
- Monitoring
- Testing in production versus staging
Ultimately with these concerns you will need to figure out what works for you based on your needs. There is no one size fits all answer to doing deployments.
What does the future hold? The answer is better tooling. As Kubernetes continues to take hold of the industry better tools will arrive on the scene. This primarily looks like service meshes such as istio and linkerd which are side car containers providing intelligent routing, out of the box support for canaries and blue/green deployments, and better monitoring.