Copy page

Features

Live Migration

Live migration moves an instance from one node to another. A running instance keeps running while it moves, so it keeps its memory and its state.

Use it to:

• Empty a node before maintenance.
• Move load from a busy node to a quieter one.

How it works

A migration copies an instance from a source node to a destination node:

• A running instance moves while it runs, and resumes on the destination with its memory and state.
• A stopped instance moves without a snapshot, and an instance in standby moves with its snapshot, so it keeps its state.
• The volumes of the instance move with it. For a running instance, the platform copies them while the instance runs, and then sends only the blocks that changed.

On the destination, the instance keeps its UUID, name, configuration, and creation time. Its uptime, start counts, CPU time, and memory-time continue from their values on the source. Its volumes and its service group move with it, and keep their UUIDs and names.

The instance gets new network interfaces on the destination. Their private IP addresses and MAC addresses come from that node, so look the instance up through its private FQDN rather than keeping its old private IP address.

Once the migration completes, the instance exists only on the destination. If a migration fails, or you abort it, the instance stays on the source node and resumes there.

Requirements

• Your account needs the migration permission on both nodes. Root users have it. Your operator grants it to other users, and should grant it only to trusted ones, because the destination creates the instance with the owner and configuration that the source sends.
• The owner of the instance has to exist on the destination node, and the instance counts against the owner's quota there, as a new instance would.
• The names of the instance, of its volumes, and of its service group have to be free on the destination.
• Both nodes need a platform release with live migration, and your operator has to set up the destination to receive migrations.

Migrating an instance

A migration takes two requests: one to the destination node, which issues a token, and one to the source node, which sends the instance.

There's no dedicated unikraft CLI subcommand yet, so the examples use the unikraft api command. In them, node-a and node-b are metros of a custom profile that point at the API endpoints of the source and destination nodes.

Issuing a token on the destination

Ask the destination node for a token for the instance:

Response
{
  "status": "success",
  "data": {
    "instances": [
      {
        "status": "success",
        "uuid": "<instance-uuid>",
        "url": "ukpmig://api.node-b.example.com:45235",
        "token": "<token>",
        "ttl_s": 300
      }
    ]
  }
}

The response holds the migration address of the destination in url, and the token. A token admits one migration, and only of the instance whose UUID you passed. Leave out uuid, or send the request without a body, to get a token that admits any one instance.

A token stays valid for ttl_s seconds until its first use. Set ttl_s in the request to choose a lifetime of up to 3600 seconds. Without it, the node's default applies, which is 300 seconds unless your operator changed it. Treat the token as a secret, since it lets the holder create an instance on the destination.

Sending the instance from the source

Pass the url and the token to the source node:

You can name the instance by name instead of uuid, or use /v1/instances/<instance-uuid>/migrate. To move more than one instance, send an array of such objects, each with a token of its own.

The request returns as soon as the platform accepts the migration, with the migration object of the instance. The source contacts the destination only when the migration starts, so a wrong or expired token doesn't fail the request. To wait for the outcome instead, set timeout_s. It takes up to the API timeout of the node, which is 60 seconds by default, and -1 waits for that maximum. When the timeout expires first, the request reports Operation timed out with the migration object, and the migration goes on. Without timeout_s, a failed migration reports no error: the migration object disappears, and the instance stays on the source.

Following a migration

While an instance migrates, its status on the source node has a migration object:

Code
"migration": {
  "destination": "ukpmig://api.node-b.example.com:45235",
  "state": "copying",
  "total_bytes": 1073741824,
  "copied_bytes": 268435456
}

The state takes these values:

Read it with unikraft api /v1/instances/<instance-uuid> --metro node-a. Only accounts with the migration permission see the migration object. During the migration, the destination lists its copy of the instance, with "incoming": true in place of destination. When the migration completes, the instance disappears from the source node.

Queued migrations start in the order of their requests. While a migration waits, you can still start, stop, and change the instance. The platform checks the instance again when the migration starts, and fails the migration if the instance no longer qualifies. If the instance is only busy at that point, the migration keeps its place in the queue and tries again later.

While the instance migrates, the platform refuses changes to it, to its service group, and to its volumes. It also refuses to suspend the instance, and to stop it without force. A forced stop or a delete of the instance aborts the migration, until the migration reaches committing. A stopped instance can still start until the destination has a complete copy, and the migration then moves it as a running instance.

A restart of the platform controller doesn't cancel a migration. The migration picks up where it stopped.

Aborting a migration

Abort a migration on the source node:

You can also use /v1/instances/<instance-uuid>/migrate/abort. The request returns at once, and the platform rolls the migration back in the background. The instance stays on the source node and resumes there. You can abort a migration in every state before committing.

Error handling

Limitations

The platform doesn't migrate the following instances yet:

• Instances created with "type": "full" (full VMs, see instance types).
• Instances with vertical scaling.
• Instances of kernel-less images.
• Instances with inline ROMs.
• Templates, and instances that are becoming one.
• Autoscale instances, and instances in a service group with autoscale.
• Instances with autokill, and instances that the platform deletes when they stop.
• Instances with dependencies, and instances that running instances depend on.
• Instances with changes to volumes, ROMs, or dependencies that wait for the next restart.
• Instances in a service group with custom certificates, or in a persistent service group.
• Instances with a custom TAP device, a relay, or autoconfig turned off on an interface.
• Instances with network interfaces on a node in managed network mode, since each of those interfaces uses a custom TAP device.

Instances cloned from a template can migrate once their template is ready.

Every volume of the instance has to qualify, or the instance doesn't migrate. A volume qualifies when it has the same owner as the instance, no other instance has it attached, and it's neither a managed volume nor a virtiofs volume.

Learn more

• Instances: how instances work, their states, and their configuration.
• Networking: reach an instance through its private FQDN.
• Services: how service groups route traffic to instances.
• Volumes: persistent storage that moves with an instance.
• The CLI reference.