Every virtual machine consists of at least two components: a kernel, and a root file system. This tutorial focuses on the latter, the rootfs (also known as initrd).
Currently, on the Unikraft Cloud platform, two types of file system archives are usable: CPIO (Copy In, Copy Out) and EROFS (Enhanced Read-Only File System).
They're packaging formats used for loading the root file system in the instance.
They consist mainly of metadata and file content without any advanced formatting to keep them as simple as possible.
The kernel loads and mounts the data from them at boot time.
Getting started
The main difference between the rootfs formats is performance-wise.
CPIO
CPIO is a simple format which translates in the need to copy and unpack the entire archive into guest memory when starting the instance.
This incurs a double memory usage during boot time, as the archive is first loaded into memory, then unpacked into a ramfs for the kernel to load.
At runtime, the guest can reclaim the initial memory allocated for the archive, and will only use the unpacked ramfs.
The boot times will still proportionally increase with the size of the CPIO archive.
Another consequence of using a CPIO initrd is that snapshots must include them, since Unikraft Cloud can't track where they're stored in guest memory.
EROFS
EROFS wins on startup efficiency by not needing to unpack the archive at all into guest memory.
This means that you don't need to accomodate for double the memory usage during boot time, and boot times are faster as well.
Another significant benefit of EROFS (although not available yet on public metros) is that Unikraft Cloud can enable DAX (Direct Access) on instances using EROFS rootfs.
This means that the host side loads the initrd, and the guest gets a map (not a copy) of it into its memory.
Thus, boot times aren't affected by the size of the rootfs at all.
Memory usage is also drastically reduced at boot, and only increases as the guest accesses files.
This also means that snapshots can exclude the original initrd.
Writing to the rootfs is still possible, because Unikraft Cloud uses overlayfs to mount the ramfs.
Any file changes are present in the snapshot.
Finally, DAX enables sharing the same rootfs image between many instances, increasing scalability.
Packaging the Rootfs
The KraftKit CLI tool supports both formats out of the box. With it, you can package a rootfs from these sources:
- a simple file
- a directory
- a Dockerfile
- a remote OCI image
If provided as a file, the CLI tool passes the file as-is, without any extra packaging.
Otherwise, you need to package the rootfs into a supported format (either CPIO or EROFS).
Already packaged archives with external tools (for example mkfs.erofs, cpio, bsdcpio) are also usable.
Right now the default format for the CLI tool is CPIO, with EROFS being an alternative, as the former is simpler, widely adoped, and better tested.
It's worth mentioning that after unpacking, there is no difference in the produced file contents between the formats.
They may only differ metadata-wise, depending on the implementation.
You may observe that a CPIO archive is typically smaller on disk than an EROFS one.
The only effect this has is on the time it takes to push the image.
Below are two attempts at creating and starting an instance from the same image, packaged using both formats.
EROFS deployment
Code
Create Instance EROFS
CPIO deployment
Code
Create Instance CPIO
The packaged archives have a size of around 750MiB each.
The EROFS-packaged instance requires at least 900MiB to work, and the CPIO one requires 2400MiB.
Simple arithmetic backs this.
In the CPIO case, the kernel gets a copy of the archive in memory and needs to remove it into ramfs.
This extraction process requires double the memory of the archive size.
This means the instance needs at least an extra 750x2=1500MiB of memory compared to EROFS to boot.
Finally, related to boot times, both cases have similar warm boot times, but the extra copies mentioned increase the cold boot time in the CPIO case.
Conclusion
Bottom-line is, using EROFS is recommended in every case where possible, as the upsides definitely outweigh the small downsides.
Learn more
- The
kraft cloudcommand-line tool reference, and in particular the deploy subcommand. - The
kraft pkgcommand reference for packaging images.