Instances
This document describes the Unikraft Cloud Instances API (v1) for managing Unikraft instances. An instance is a MicroVM running a single instance of your app.
Drive instances from your own code
Everything on this page is in the SDKs, so instances are something your program creates and drives directly:
Code
See the JavaScript, Python and Go SDKs.
Instance states
An instance can be in one of the following states:
| State | Description |
|---|---|
stopped | The instance isn't running and doesn't count against live resource quotas. Connections can't establish. |
starting | The instance is booting up. This typically takes just a few milliseconds. |
running | Your app reached its main entry point. |
draining | The instance is draining connections before shutting down. No new connections can establish. |
stopping | The instance is shutting down. |
standby | The instance has scaled-to-zero. The instance isn't running, but will be automatically started when there are incoming requests. |
Unikraft Cloud reports these as instance state values via the endpoints.
You can visualize these states with the CLI.
Here is an example of an instance that's in stopped state:
Code
Code
You can retrieve even more instance details via the API:
GET /instances/bd281cc4-7640-4cbc-af48-5a44c2bd7b74
Stop reason
To understand why Unikraft Cloud stopped an instance or is shutting it down, it provides information about the stop reason.
You can retrieve this information via the GET /instances endpoint when an instance is in the draining, stopping, stopped or standby state.
The stop_reason contains a bitmask that tells you the origin of the shutdown:
| Bit | 5 [Z] | 4 [F] | 3 [U] | 2 [P] | 1 [A] | 0 [K] |
|---|---|---|---|---|---|---|
| Desc. | Stop due to scale-to-zero | This was a force stop1 | Stop initiated by user2 | Stop initiated by platform3 | App exited - exit_code available | Kernel exited - stop_code available |
For example, the stop_reason has the following values in these scenarios:
| Value | Bitmask | Scenario |
|---|---|---|
39 | 100111/Z--PAK | Scale-to-zero shutdown. The platform initiated the stop because the instance scaled to zero, and both the app and kernel exited. |
31 | 011111/-FUPAK | Forced user-initiated shutdown where the app and kernel still reported clean exits before the stop completed. |
28 | 011100/-FUP-- | Forced user-initiated shutdown without app or kernel exit information. |
15 | 001111/--UPAK | Regular user-initiated shutdown. The app and kernel have exited. The exit_code and stop_code show if the app and kernel shut down cleanly. |
13 | 001101/--UP-K | The user initiated a shutdown but the app was forcefully killed by the kernel during shutdown. This can happen if the image doesn't support a clean app exit or the app crashed after receiving a termination signal. In this case, ignore exit_code. |
7 | 000111/---PAK | Platform-initiated shutdown. The app and kernel have exited. The exit_code and stop_code show if they shut down cleanly. |
3 | 000011/----AK | The app exited. The exit_code and stop_code show if the app and kernel shut down cleanly. |
1 | 000001/-----K | The instance likely experienced a fatal crash and the stop_code contains more information about the cause of the crash. |
0 | 000000/------ | The stop reason is unknown. |
There can be a short delay of a few milliseconds between the instance reaching the stopped state and Unikraft Cloud updating the stop_reason (or vice versa).
The scale-to-zero shutdown is a special case where the platform initiates the stop because the instance scaled to zero, and both the app and kernel exited.
In the API, the instance appears in the standby state, instead of stopped.
Exitcode
The app exit code is what the app returns upon leaving its main entry point.
The encoding of the exit_code is app specific.
See the documentation of the app for more details.
An exit_code of 0 indicates success or no failure.
Stopcode
If bit 0 [K] is set in stop_reason, stop_code contains the 32-bit code reported by the guest kernel.
Unikraft Cloud also uses platform-defined values when the stop originates in the platform before the guest kernel can report its own code.
Currently the platform-defined values are:
| Value | Symbol | Scenario |
|---|---|---|
0 | UNKNOWN | No platform-specific stop code is available. |
1 | NOMEDIUM | The instance couldn't start because the image or another required medium wasn't available, for example because an image pull failed. |
When the guest kernel reports a stop code, it uses the following encoding:
| Bits | 31 - 24 (8 bits) | 23 - 16 (8 bits) | 15 [T] | 14 - 8 (7 bits) | 7 - 0 (8 bits) |
|---|---|---|---|---|---|
| Desc. | Reserved4 | errno | shutdown_bit | initlvl | reason |
Reason
The reason can be any of the following values:
| Value | Symbol | Scenario |
|---|---|---|
0 | OK | Successful shutdown. |
1 | EXP | The system detected an invalid state and actively stopped execution to prevent data corruption. |
2 | MATH | An arithmetic CPU error (for example, division by zero). |
3 | INVLOP | Invalid CPU instruction or instruction error (for example, wrong operand alignment). |
4 | PGFAULT | Page fault - see errno for further details. |
5 | SEGFAULT | Segmentation fault. |
6 | HWERR | Hardware error. |
7 | SECERR | Security violation (for example, violation of memory access protections). |
A reason of 0 indicates a clean shutdown.
Ignore the other bits of stop_code when checking for a crash.
Init level
initlvl indicates the initialization or shutdown phase at stop time.
A level of 127 means the instance was executing the app.
Shutdown bit
shutdown_bit indicates the system was shutting down.
Error number
errno is a Linux error code number that provides more detail about the root cause.
For example, an out-of-memory (OOM) situation triggers a page fault PGFAULT(4) with errno set to ENOMEM(12).
In that case the stop_code is 0x000C7F04=818948 and the stop_reason is -----K (1) if the stop occurred during app execution.
By default, a guest that runs out of memory stops the whole instance instead of killing a single process.
The workload then doesn't continue in a partial state.
A process can opt in to the out-of-memory killer by setting its /proc/self/oom_score_adj to 0.
The kernel may then kill that process alone and keep the instance running, so the rest of the workload has to cope with losing it.
Restart policy
When an instance stops because the app exits or crashes, Unikraft Cloud can restart it automatically according to the restart policy. The policy can have the following values:
| Policy | Description |
|---|---|
never | Never restart the instance (default). |
always | Always restart the instance when Unikraft Cloud initiates the stop from within the instance (that is, the app exits or the instance crashes). |
on-failure | Only restart the instance if it crashes. |
When an instance stops, Unikraft Cloud evaluates the stop reason and the restart policy to decide whether to restart. It uses an exponential back-off delay (immediate, 5s, 10s, 20s, 40s, 5m) to slow down restarts in tight crash loops. If an instance runs without problems for 10s, Unikraft Cloud resets the back-off delay and the restart sequence ends.
The restart.attempt value in GET /instances counts restarts in the current sequence.
The number of completed restarts is restart_count.
The restart.next_at field indicates when the next restart occurs if a back-off delay is in effect.
A manual start or stop of the instance aborts the restart sequence and resets the back-off delay.
Creating instances
Instance types
Every instance runs as one of two virtual machine types, which you pick with --type:
| Type | Description |
|---|---|
micro | The default, running on Firecracker. |
full | Runs on QEMU, which GPU support requires, and needs a plan with full VM support. |
Full VMs running on QEMU are required for GPU support.
They don't support branching, checkpointing, or vertical scaling yet, and work only with block-based volumes rather than virtiofs.
A create request therefore can't combine --type=full with --branch or --checkpoint.
Replicas
The POST /instances request accepts a replicas field (default 0) that creates more copies of the instance alongside the base one.
| Value | Instances created |
|---|---|
0 | 1 (the base instance) |
1 | 2 (base + 1 replica) |
N | N + 1 |
All instances share the same image, memory, arguments, and service group configuration. Replicas receive independent names and UUIDs.
You can do this with the CLI as well:
Code
Vertical scaling
An instance can grow and shrink its memory and vCPUs while it runs.
Opt in by setting a range in the create request: min_memory_mb and max_memory_mb for memory, min_vcpus and max_vcpus for vCPUs.
The instance boots with the minimum, and the platform adjusts the size within the range based on the instance's memory utilization and CPU pressure.
See Vertical scaling for the details, requirements, and limitations.
GPUs
Limited Access
GPU support is currently only available as a preview for enterprise customers. To try out the preview, reach out to the Unikraft Cloud Discord or send an email to support@unikraft.com. The interface described here reflects the current implementation and may change before general availability.
Instances can run with an NVIDIA GPU attached. Depending on the host, that GPU is either a whole physical GPU or a vGPU. A vGPU is a virtual GPU that the host carves out of a physical GPU, so more than one instance shares the physical GPU. A vGPU reports the same model as the physical GPU it comes from, and your workload uses it like any other GPU.
Pass --gpus when you create the instance, together with the full instance type that GPUs require:
Code
An instance can currently hold at most one GPU, and you can't choose which one. Build the workload image on a glibc-based distribution rather than Alpine, because the NVIDIA user-space libraries don't run on musl.
GPU allocation
The platform allocates a GPU when the instance starts, not when you create it, and this includes instances created with autostart. On the first start, it picks any free GPU on the host and records that GPU's UUID and model on the instance. If no GPU is free, the start fails with an error stating that no GPU matching the instance is currently available.
The instance holds the GPU only while it runs. Stopping the instance, or scaling it to zero, returns the GPU to the host's pool so another instance can use it in the meantime. On every later start, the platform first tries to reacquire the same GPU, and falls back to any free GPU of the same model if another instance holds that one. It never assigns a GPU of a different model, because a resumed guest expects the same hardware as before the suspend. To reserve GPU capacity for a set of instances, limit how many of them run at the same time.
To see the GPU an instance holds, inspect it:
Code
The gpus.*.allocated field reports whether the instance holds a GPU right now.
The gpus.*.uuid and gpus.*.model fields identify that GPU.
The model is the GPU's vendor and device ID in hexadecimal form, such as 10de:1eb8 for an NVIDIA Tesla T4.
A stopped instance holds no GPU, so it reports allocated as false and no UUID, but keeps the model it last ran on.
resources.gpus reports how many GPUs the instance requests.
Listings sort and filter on these fields like any other field.
Scale-to-zero and templates for GPU instances
GPU instances backed by a vGPU support scale-to-zero, suspend and resume, and instance templates. Suspending a GPU instance requires a GPU that can save and restore its device state, which vGPUs support.
When a vGPU instance scales to zero, the platform snapshots the VM together with the state of its GPU, detaches the GPU, and returns it to the pool. On the next request, the platform reacquires a GPU as described in GPU allocation and resumes the VM from the snapshot with the GPU attached. This lets a host serve more GPU instances than it has GPUs. An idle instance releases its GPU, and a woken instance resumes on any free GPU of the same model.
A GPU accesses guest memory directly and can't wait for a page the platform hasn't loaded yet. Resuming a GPU instance loads its whole memory from the snapshot before the VM continues, so resume time grows with the instance's memory size. CPU-only instances load memory on demand instead.
Instances created from a template of a GPU instance inherit its GPU configuration and model.
A create request that uses such a template can't set --gpus.
The value comes from the template.
Wait for running
By default POST /instances returns as soon as the platform queues the instance.
Set timeout_s non-zero to block until the instance reaches the running state or the timeout expires.
For example:
POST /instances
wait_timeout_ms is a deprecated compatibility field.
When set, the platform rounds the value up to the next full second.
Use timeout_s instead.
The unikraft CLI automatically waits for the instance to be running after creation, up until a preconfigured timeout of 10 seconds.
To wait for an existing instance, pass query parameters to the wait endpoint:
Code
Delete on stop
Pass delete-on-stop in the features array to automatically delete the instance when it stops:
POST /instances
This is useful for ephemeral workloads—batch jobs, one-shot tasks—where you don't need to keep the stopped instance.
You can also use the CLI to set this up:
Code
To use the "delete-on-stop" feature, set the restart_policy to never.
Nested virtualization
Limited Access
Nested virtualization is available as part of enterprise plans. To enable it for your account, reach out to the Unikraft Cloud Discord or send an email to support@unikraft.com.
An instance runs without the host's virtualization extensions by default, since the platform hides them from the guest.
Pass nested-virt in the features array to expose them, so that a workload shipping its own virtual machines runs as an ordinary instance:
POST /instances
You can also use the CLI to set this up:
Code
The guest kernel needs the matching driver to use the extensions, and exposes /dev/kvm once the driver is active.
On x86_64, a Linux guest needs CONFIG_KVM together with CONFIG_KVM_INTEL or CONFIG_KVM_AMD.
On arm64, a Linux guest needs CONFIG_KVM, which the kernel builds in rather than loading as a module.
The host must support nested virtualization itself.
An instance with nested-virt doesn't start on a host without this support.
A create request that starts from a snapshot—an instance template, a branch, or a checkpoint—can't add nested-virt.
Set the feature on the instance that you snapshot instead, and every instance that starts from that snapshot inherits it.
Stopping instances
Drain timeout
By default PUT /instances/stop stops the instance immediately.
Set drain_timeout_ms to allow the instance to finish serving in-flight connections before it stops.
| Value | Behaviour |
|---|---|
0 | Stop immediately (default). |
-1 | Use the platform's maximum drain timeout. |
N | Drain for up to N milliseconds, then stop. |
PUT /instances/stop
You can also use the CLI to set this up:
Code
While draining, the instance enters the draining state.
The platform accepts no new connections.
The instance stops once all connections close or the timeout elapses.
You can't combine drain_timeout_ms with force: true.
Setting both returns a 400 error.
Instance templates
An instance template is a snapshotted instance that acts as a source for cloning new instances. Cloning a template creates a new instance that resumes from the exact original system state. It preserves memory contents, open files, and populated caches to bypass the standard boot sequence. Once you convert an instance into a template, you can't reverse the process.
To transition an actively running instance into a template, from the guest write the value 1 to /uk/libukp/template_instance:
Convert instance to template
This action instructs the controller to freeze running processes and save the instance as a reusable template. The write blocks until the platform has snapshotted the instance and created the new instance from the resulting template. When the write returns, the guest is already running as the freshly cloned instance, restored from the snapshot rather than continuing as the original.
This lets the guest finish any instance-specific setup—such as reading per-instance data from an attached inline ROM—immediately after the write completes.
If the template already exists, the clone happens right away and the write returns without re-snapshotting.
You can also create a template from an instance (even when it's running) by calling POST /instances/templates, or using the CLI:
Code
Code
You can list templates with GET /instances/templates or the CLI:
Code
Code
Now, you can create an instance from the template using the unikraft CLI:
Code
Code
Templates support delete locks, tags, and autokill. The platform measures autokill on a template from the time of last clone, not from instance stop time.
You can also have the platform create a template implicitly as part of an instance-create call, instead of converting and managing it yourself. See On-demand templates.
Updating a template
You can move a template to new images without recreating it.
A PATCH /instances/templates request changes the image of a template, and the images of its ROMs and plugins:
Code
These properties only take the set operation.
The patches for one template in a request make up a single update, and if one of them fails, none of them applies.
The platform boots a new revision of the template with the new images, and turns it into a template the same way as the first revision.
Once the new revision is ready, it takes over the UUID and the name of the template.
While the update runs, the status of the template lists it in updates as pending.
If the new revision fails, the template doesn't change, and updates reports the update as failed, with the reason in error.
The instances of the template then follow the update:
- An instance created from the template uses the new revision at its next start, unless it has a snapshot of its own. This includes instances that scale to zero in stateless mode.
- An instance with a snapshot of its own, for example a suspended one, keeps its revision as long as it has that snapshot.
- Templates created from the updated template follow it, and autoscale uses the new revision for new instances.
- An instance with an image of its own, or with ROMs or plugins that replace those of the template, keeps its revision. That holds whether you set them at creation or later.
The status of an instance created from a template names its source in clone_source.
Note the following:
- You can't update a template with volumes, or a template without exactly one network interface.
- A template takes one update at a time. A request for a template whose update still runs fails.
- An update of the ROMs or plugins must list every ROM or plugin of the template, except those inherited from a parent template, with the same names, paths, and authorization. Only their images and the plugin configuration change.
- A template created from another template changes its image only together with its parent.
Nested templates
You can clone a template from another template, creating a hierarchy. Each clone inherits the parent's full state at clone time and can independently accumulate further state before you convert it into its own template. This enables layered configurations, for example a base template with a warm database connection pool, and child templates specialized for different query workloads.
No depth limit applies to nesting, and circular references aren't possible because the template state is immutable once set.
You can also build such a hierarchy in a single API call with on-demand templates, where the platform creates each level only if it doesn't already exist.
For more information, check out the API reference for instance templates here.
Instance metrics
Use the GET /instances/metrics endpoint to retrieve runtime statistics for one or more instances.
With no identifiers, the platform returns metrics for all instances in your account.
For more information, check out this dedicated section on instance metrics.
Instance logs
Use the GET /instances/logs endpoint to retrieve logs for one or more instances.
The logs capture the instance's stdout and stderr output, and they're preserved across restarts and stops.
Read endpoints accept query parameters as an alternative to a JSON body. For example, to read the last 4 KiB of an instance's log:
Code
A request that sends a body can't also give query parameters, including uuid or name.
With the CLI, you can also follow the logs in real time:
Code
Learn more
- The CLI reference.
- Unikraft Cloud's REST API reference, in particular the section on instances.
Footnotes
-
A forced stop doesn't give the instance a chance to perform a clean shutdown. Often,
exit_codeandstop_codeare therefore not available. If the guest reports an exit while the forced shutdown is in progress, bits 0 [K] and 1 [A] can still be set. ↩ -
A stop command originating from the user travels through the platform controller. This is why bit 2 [P] will also always occur for user-initiated stops. ↩
-
Internally this bit is set when ukpd, the platform controller, initiates the stop. ↩
-
The system sets reserved bits to 0; ignore them. ↩