Troubleshooting

This guide explains how to debug apps on Unikraft Cloud so you can fix issues or collect enough information for the support team to help you. It follows platform best practices and supports efficient troubleshooting. If you need help, reach out to Unikraft Cloud Support on Discord or write to support@unikraft.cloud.

Debugging running apps

The most direct way to debug an app is to use the app console output, which may include kernel output. To see it, after starting the Unikraft Cloud instance, use:

Code
unikraft instances logs <instance_name>

In case of a crash, you'll see a full crash output:

Code
Powered by Unikraft Telesto (0.16.2~9c264902)
[    0.066949] CRIT: [libukvmem] Cannot handle write page fault at 0x1000bb8024 (ec: 0x2): -12
[    0.067680] CRIT: [libkvmplat] page fault handler returned error: -12
[    0.068227] CRIT: [libkvmplat] Unhandled page fault vaddr=0x1000bb8024, error code=0x2
[    0.068910] CRIT: [appelfloader] Unikraft crash - Telesto (0.16.2~9c264902)
[    0.069503] CRIT: [appelfloader] Thread "python3"@0x4017e4020
[    0.069993] CRIT: [appelfloader] RIP: 0008:000000100041fdcb
[    0.070468] CRIT: [appelfloader] RSP: 0010:000000100017f760 EFLAGS: 00010206 ORIG_RAX: 0000000000000002
[    0.071269] CRIT: [appelfloader] RAX: 0000001000bb8000 RBX: 00000004018012f0 RCX:0000000000000007
[    0.072040] CRIT: [appelfloader] RDX: 0000000000000000 RSI: aaaaaaaaaaaaaaab RDI:00000010007bccc0
[    0.072793] CRIT: [appelfloader] RBP: 000000000000001a R08: 0000000000000001 R09:00000010004e2b99
[    0.073558] CRIT: [appelfloader] R10: bb9e744ce6503a27 R11: 0000000401861290 R12:00000010007bccc0
[    0.074312] CRIT: [appelfloader] R13: 00000010007bbd48 R14: 000000100017f890 R15:0000000000000038
[    0.075076] CRIT: [appelfloader] Stack:
[    0.075406] CRIT: [appelfloader]  100017f760  00 00 00 00 00 00 00 00  |........|
[    0.076043] CRIT: [appelfloader]  100017f768  a1 01 00 00 00 00 00 00  |........|
[    0.076680] CRIT: [appelfloader]  100017f770  48 bd 7b 00 10 00 00 00  |H.{.....|
[    0.077324] CRIT: [appelfloader]  100017f778  e0 92 7a 00 10 00 00 00  |..z.....|
[    0.077959] CRIT: [appelfloader]  100017f780  48 bd 7b 00 10 00 00 00  |H.{.....|
[    0.078596] CRIT: [appelfloader]  100017f788  90 f8 17 00 10 00 00 00  |........|
[    0.079233] CRIT: [appelfloader]  100017f790  38 00 00 00 00 00 00 00  |8.......|
[    0.079870] CRIT: [appelfloader]  100017f798  72 fc 41 00 10 00 00 00  |r.A.....|
[    0.080504] CRIT: [appelfloader] Call Trace:
[    0.080871] CRIT: [appelfloader]  [0x000000100041fdcb]
[    0.081311] CRIT: [appelfloader]  Bad frame pointer

or even

Code
[    0.850992] Initramfs unpacking failed: write error

Inspect the instance details to see the stop reason:

Code
unikraft instances get <instance_name>

This yields output like:

Code
metro:           fra
name:            node-playwright-chromium-nwa95
uuid:            59cd6c94-611c-4ca8-911f-78c7db95ad12
state:           stopped
image:           <my-org>/node-playwright-chromium
resources:
  memory:        2.441GiB
  vcpus:         1
service:
  name:          divine-dust-73sxgnb6
  uuid:          3c7ce2b4-ae86-4a21-8562-55033d5409d5
  domains:
  - fqdn:        divine-dust-73sxgnb6.fra0-fe-test.unikraft.app
networks:
- uuid:          62982265-6521-401c-878a-d7e31804c63c
  private-ip:    10.0.0.109
  mac:           12:b0:0a:00:00:6d
timestamps:
  created:       2 days ago
scale-to-zero:
  enabled:       true
  policy:        on
  cooldown-time: 1s
stop:
  reason:        kernel crash: out of memory (ENOMEM)

This output shows the stop reason. Here the cause is insufficient memory.

If the stop reason lacks detail, enable debug tracing for the instance.

To do that, pick an app from the examples repository or an app directory you created. Update the runtime entry in the Kraftfile to reference the debug build by adding -dbg to the runtime name. For example, if you want to run the httpserver-go1.21 example with debug output, update its Kraftfile as follows:

Code
spec: v0.7

runtime: base-compat:latest-dbg

targets:
  - kraftcloud/x86_64

rootfs:
  source: ./Dockerfile
  format: erofs

cmd: ["/server"]

That is, change base-compat:latest to base-compat:latest-dbg.

Now you're ready to re-deploy:

Code
unikraft build . --output <my-org>/http-go-strace:latest
unikraft run --metro fra \
  --name http-go-strace \
  -p 443:8080/http+tls \
  --image <my-org>/http-go-strace:latest

You can now inspect the logs as before and view system call tracing:

Code
unikraft instances logs http-go-strace

This mechanism works for all apps and runtimes. Contact Unikraft Cloud on the Discord server and include this output if you need more detail.

Code
unikraft run --metro fra --restart on-failure --image my-app:latest

Debugging the build and push steps

The CLI performs steps on your local device before actually deploying your app to Unikraft Cloud:

1. Downloads the runtime image (defined in the Kraftfile) from the Unikraft Cloud registry.
2. Builds the app filesystem using the Dockerfile via BuildKit.
3. Packages the app filesystem and the runtime in an Open Container Initiative (OCI) image.
4. Pushes the OCI image to the to the Unikraft Cloud registry under your username's namespace.

If any of these steps fail, enable CLI debugging with the --log-level and --log-type flags:

Code
unikraft build --log-level debug --log-type text ...

You should then see debug output for the 4 steps above. This output should give enough context to diagnose the issue.

Caching the app's filesystem for faster builds

When using a Dockerfile for the app filesystem, Unikraft Cloud passes the commands to BuildKit.

The unikraft CLI uses the first BuildKit builder that it finds, in this order:

1. The builder at the address in the BUILDKIT_HOST environment variable.
2. The builder on the default BuildKit socket.
3. The builder in Docker.

The builder keeps its cache between builds. To control where the cache stays, start a BuildKit container and set BUILDKIT_HOST to its address:

Code
docker run -d --name buildkitd --privileged moby/buildkit:latest
export BUILDKIT_HOST=docker-container://buildkitd

The approach above caches builds in the BuildKit container filesystem. Another approach saves the cache in a local host directory:

Code
docker run -d --name buildkitd --privileged -v $HOME/.buildkit-cache:/var/lib/buildkit moby/buildkit:latest
export BUILDKIT_HOST=docker-container://buildkitd

Here $HOME/.buildkit-cache is a local path on your machine where BuildKit stores cache data.

Large filesystem build process gets stuck

When the filesystem is larger than about 800 MB, the build may get stuck. A limitation in BuildKit, the current filesystem build component, causes the issue.

Work around this by reducing the filesystem image below 800 MB. Work is in progress to integrate a component without this limitation.

Gateway error when accessing a service

When you query a Unikraft Cloud service via its public address, such as:

Code
curl https://frosty-bobo-zeev783o.fra.unikraft.app

you may get a Bad Gateway response:

Code
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>openresty</center>
</body>
</html>

This happens when you specify the wrong internal app port (for example, the app exposes port 8080 but you use 443:80 instead of 443:8080).

Another error response you may get is:

Code
<!doctype html>
<html lang="en">
<head>
        <meta charset="UTF-8">
        <meta name="viewport"
              content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
        <meta http-equiv="X-UA-Compatible" content="ie=edge">
        <title>Service not found</title>
</head>
<body>
There is no service on this URL.
</body>
</html>

This happens when you connect with an HTTPS client (port 443) and the app doesn't expose that port. Database services such as MongoDB or MariaDB use different ports (for example, 27017, 3306).

Use the correct exposed port. You may need a TLS tunnel (see below).

If you still have issues, it might mean that the app fails to start or crashes immediately. In that case, inspect the app logs first.

Connecting to a non-TLS app

Unikraft Cloud uses TLS to expose services to the outside world. Some apps (such as MongoDB or MariaDB) don't use TLS.

Create a TLS tunnel with the unikraft instances tunnel command or with socat. Both open a local endpoint (localhost / 127.0.0.1) and forward the traffic over TLS. See the MariaDB guide for an example.

Launched app not visible in list

The most common reason is that you deployed an app to one metro but listed a different one. Use the --metro flag to select the metro for each command:

Code
unikraft instances list --metro <metro_name>

Learn more

• The CLI reference.
• Unikraft Cloud's REST API reference.
• Many more guides here.