Annotations
Annotations attach arbitrary key-value metadata to an instance. They use the same key syntax as Kubernetes annotations, so you can reuse your existing keys and tooling.
Unlike tags, which act as controller-level labels that only the platform sees, annotations also reach the guest. You can use them for three things:
- Custom metadata—the instance's startdata includes its annotations, so the guest can read them at runtime.
- Guest network configuration—the guest applies the CNI result in the
unikraft.com/cniannotation to its network interfaces. - Structured log output—you select which annotations the platform injects into the VM's console log output.
Dedicated unikraft CLI subcommands for annotations are coming soon.
In the meantime, drive annotations through the API, which you can invoke with curl or the unikraft api command.
Annotation keys
An annotation is a set of key-value pairs.
A value is any string without ASCII control characters (0x00–0x1f and 0x7f, which covers tab, newline and carriage return), and multi-byte UTF-8 goes through fine.
The API rejects a value carrying an ASCII control character with 400 Bad Request.
Keys follow the Kubernetes annotation key syntax:
- A key can be a plain name (
my-key) or carry an optional DNS prefix separated by a slash (example.com/annotation2). - Each key needs a name segment of at most 63 characters, containing alphanumerics,
-,_, and., and beginning and ending with an alphanumeric character. - The optional prefix is a DNS subdomain (a series of DNS labels joined by
., at most 253 characters) followed by a/. It can't be a wildcard domain.
An instance holds at most 256 annotations.
For example:
Code
Creating an instance with annotations
Add an annotations object to the instance body when you create it.
Each entry is a key-value pair:
Code
Annotation keys are Kubernetes-compatible, so they can be plain keys (my-key) or prefixed with a domain (example.com/annotation2).
Reading annotations from the guest
The platform includes the instance's annotations in its startdata, which the guest reads from /sys/class/uio/uio0/device/startdata:
Code
Code
Your app parses this JSON and reads the annotations object like any other startdata field.
Guest network configuration with CNI
unikraft.com/cni is a reserved annotation.
Its value is a CNI Result document, and the guest applies the addresses it holds to its own network interfaces.
This hands the addressing of an instance to an external CNI plugin or IPAM system. The plugin decides the addresses, writes its result into the annotation, and the guest configures itself from it. Your image needs no agent for it and no boot script of its own.
The annotation pairs with an interface without an address, which reaches the guest as a bare link on a TAP device of your own. See Kubernetes for the CNI integration that Kraftlet drives on the cluster side.
The annotation value is the document as a JSON string, so the JSON inside it carries escapes:
Code
That instance keeps a pool-allocated primary interface and adds a second one on the vpc0 TAP device, with a MAC address you pick and no IP address.
Unescaped, the annotation holds a standard CNI Result (types/100):
Code
The guest gives 192.168.1.5/24 to the link whose MAC address is 0a:58:c0:a8:01:05, and brings that link up.
Writing the document
Give every address a MAC address to land on:
- Each
ips[]entry points through itsinterfaceindex at an entry of the document's owninterfaces[]array, not at the instance'snetwork_interfaces. An entry with nointerfacefield uses index0. - That
interfaces[]entry supplies the MAC address of the interface to configure, sointerfaces[].namecan say anything. Without a MAC address to match, the index counts the instance's non-loopback interfaces from0instead. The entry carries nomac, the document holds nointerfaces[]array, or the index sits past the end of that array. - Set the MAC address yourself when you create the instance, or read it from the instance status.
- Addresses take CIDR notation, IPv4 or IPv6, and the prefix length is mandatory.
- The document needs a
cniVersionof0.3.1,0.4.0,1.0.0, or1.1.0, and has to be a result rather than a network configuration. - Keep the document on one line, because annotation values reject control characters.
Only interfaces[].mac, ips[].interface, and ips[].address have any effect.
routes, gateway, dns, mtu, and sandbox do nothing, so you can pass a plugin's full result through untouched.
Set the default gateway and the resolver with the instance's own gateway and nameserver fields.
Changing addresses
Running instances pick up updates to their annotations, and reconcile them against what's currently configured, so an address change doesn't require a restart.
When nothing happens
A single MAC address that matches no interface stops the whole document from applying, leaving every interface untouched. Check that each MAC address in the document matches one on the instance, then read the instance logs, which name any document that fails to parse or apply.
Patching annotations
Update the annotations on an existing instance with a PATCH /instances request.
The body is an array of patch operations, each naming the target instance, the annotations property, and one of three operations: set, add, or del.
Annotation changes apply only while the instance is in the stopped state.
If you patch a running instance, the platform accepts the request and queues the change, then applies it the next time the instance stops.
Replace all annotations (set)
set replaces all existing annotations with the object you provide:
Code
Add or update annotations (add)
add merges the given annotations into the existing set.
It overwrites keys that already exist and leaves the other keys untouched:
Code
Delete annotations (del)
del removes specific keys.
The value is an array of key names (or a single key as a string):
Code
Annotations in log output
Beyond the guest, the platform can inject selected annotations into a VM's console log output. This helps with structured logging to an external collector.
This forwarding happens at the node level rather than through the instances API, so coordinate with Unikraft to enable it for your deployment. The platform exposes two options:
Code
--vmm-console-portsdefines named console ports as<name>:<type>[/<format>][:<path>]. Hereappandapp2aresocketports using thejsonformat, each writing to a Vector socket.--vmm-console-annotationsselects which annotations appear in each port's output, as<port>:<key1>+<key2>,...or<port>:*:app:my-key+test.unikraft.com/my-annotation—forward only the listed annotation keys (joined with+) to portapp.app2:*—the*wildcard forwards all annotations to portapp2.
When the guest writes to the app console (for example, echo "blub" > /dev/vport2p1), the collector receives the message together with the selected annotations:
Code
Annotation events
The platform's event log carries a vm.annotate event, which it emits whenever an instance's annotations change:
Code
An external system consumes the event to keep track of the instance's annotation values, without polling the instance API. The event also gives log processing a way to map an instance UUID back to its annotations.
Limitations
- An instance holds at most 256 annotations.
- Annotation keys must follow the Kubernetes key syntax, and values are strings with no ASCII control characters in them.
- A patched annotation change applies once the instance reaches the
stoppedstate. - A
unikraft.com/cnidocument holds at most eight interfaces and 16 addresses, and anips[].interfaceindex stays below eight. - The guest applies only
interfaces[].mac,ips[].interfaceandips[].addressfrom that document, and needs a recent base-compat guest kernel to apply it at all. - Dedicated
unikraftCLI subcommands aren't available yet, so drive annotations through the API. - The platform sets up log-output forwarding at the node level, not through the instances API.
- The
vm.annotateevent needs switching on in the node's event configuration.
Learn more
- Tags: controller-level labels for organizing platform resources.
- Custom network configuration: bring your own interfaces, MAC addresses and TAP devices, including interfaces without an address for a CNI plugin to fill in.
- Kubernetes: how Kraftlet runs the CNI plugins of your cluster for Unikraft Cloud instances.
- Instances: create and manage instances.
- Unikraft Cloud's REST API reference, in particular the section on instances.