Live Migration, now live at Unikraft

In the cloud, every workload/VM is assigned to hardware when it’s created. A scheduler picks a node and from then on the two are inextricably linked until the workload finishes and the VM can be destroyed. If the node needs patching, runs hot, gets expensive or starts to fail, the only way to separate them is to stop the workload.

Live migration breaks that link. A running microVM, with its memory, CPU state, network identity and attached volumes can move to another node and carry on. Both the workload and the user won’t know this has happened.

This is a fundamental shift: Where a workload runs no longer needs to be decided at creation-time, you can now decide and change this at any time, as often as you like, with better information, as conditions change, etc. This brings an unprecedented level of operational and budgetary flexibility.

Once placement can change at runtime instead of being fixed at creation, a lot of old problems turn into simple moves. Since this is a new paradigm, we wanted to share a few ways to use this for saving time & money, and improving stability.

Live maintenance

Think of a kernel patch, a firmware update, or a NIC that needs reseating. Until now, a node going down for maintenance meant cordoning it and waiting for its workloads to drain off, which for long-lived connections can take hours. If you’re running BYOC/renting hardware or instances, this is even worse as you have 0 control as to whether, or when, or for how long, this downtime happens. You’re at the mercy of your provider and your client’s workloads in terms of having enough time to drain everything off of that node. With live migration you can move everything off in one go. If you’re the one operating the node then you can decide exactly when to take that node offline for maintenance.

Hardware failure warnings

Disks and DIMMs usually announce they’re dying before they die with things like rising SMART counts or corrected ECC errors. Instead of just logging those events, you can now use them as triggers to act. Move the instances off the failing host while it still works, then pull it.

Moving to newer hardware

A new server generation lands in the rack. Faster cores, more memory per dollar. Getting your fleet onto it no longer needs a roll out and workload draining project, not to mention maintenance windows and customer interaction. With live migration instances move across while they run, and the old generation retires on your schedule.

Region and data-centre moves

Because volumes travel with the instance, a microVM isn’t tied to the site it was created in. Move it closer to the users it’s serving, or empty out a facility that’s closing, without a cutover weekend.

Rebalancing, and OOM prevention

Placement decisions are usually made at start, but conditions shift almost always. Only a few hours in, one node is running hot and the one next to it is idle, so live migration lets you correct the placement after the fact. This clearly applies to exhausting all CPU cycles on a host, but equally to memory, where a VM may need more memory to run. In a standard scenario, the VM may OOM, but with live migration, it may simply trigger a migration to another host and keep running without issues.

Noisy neighbours

Strong isolation keeps a neighbour out of your memory, but it can’t fully stop one from saturating the shared disk or memory bandwidth on the host. When a latency-sensitive instance starts to feel it, live migration allows you to move that instance somewhere quiet.

Instant scale-in

Scaling out is the easy half. After the spike, the autoscaler cordons nodes to scale back in, and then waits. There’s always a long-lived session or a slow job that keeps the node alive, and you pay for the whole machine to host one straggler.

Spot instances for critical workloads

Spot capacity is far cheaper than on-demand, but of course the provider can take it back on very short notice (for example two minutes on AWS) and so people usually don’t run critical workloads this way. Live migration lets you take the interruption notice to trigger the move immediately.

Packing for cost

Overnight, traffic drops and instances end up scattered thinly across the fleet. Consolidate them onto fewer nodes during the quiet hours and power the empty ones down. When the morning ramp arrives, spread them back out.

Overall, live migration lets you do one thing: Cut the link between a workload and the machine it started on. Now you can run your workloads where it makes sense, at any time, completely invisible to the workload.

Next: live migration on Kubernetes

In Kubernetes, a pod lives and dies on the node it was scheduled to. Evict it and it’s terminated, then a replacement starts somewhere else. Every scenario above is handled the same way: drain, kill, reschedule, and absorb the restart.

We’re bringing live migration to Kubernetes, so that moving a workload off a node means moving it with full state, not restarting it.