Load Balancing

Load balancing in Unikraft Cloud is easy: as soon as you attach more than one instance to a service, Unikraft Cloud will automatically start balancing traffic between many instances. The load balancing happens based on the number of connections (in TCP mode) or requests (in HTTP mode). More on that below.

Because of load balancing, instances in a service must be of the same kind (for example, expose the same port). You can remove instances from a service at any time, and, when you do, Unikraft Cloud will immediately take the instance out of the load balancing service.

Soft and hard limits

Services have soft and hard limits for the number of concurrent requests and connections. The limits apply per instance. For HTTP services (that is, using the http handler), the system checks each individual in-flight request against the limit, but not the underlying TCP connection. For TCP services, the system counts the individual open connections. In the following, the term request refers to both requests and connections.

The load balancer uses the soft limit to decide when to wake up another standby instance. For example, if you set the soft limit to 5 and the service consists of 2 standby instances, one of the instances can receive up to 5 concurrent requests. The 6th parallel request wakes up the second instance. If there are no more standby instances to wake up, the number of requests assigned to each instance will exceed the soft limit. The load balancer ensures that when the number of in-flight requests goes down again, instances go into standby as fast as possible.

The hard limit defines the max number of concurrent requests that an instance can handle. The load balancer will never assign more requests to a single instance. In case there are no other instances available, the load balancer blocks excess requests (that is, they're blocked and not queued).

Setup

To set load balancing up, first use unikraft run with the -p flag to create the service as part of the instance creation. For example, use NGINX as the app:

Code
git clone https://github.com/unikraft-cloud/examples
cd examples/nginx/
unikraft build . --output <my-org>/nginx:latest
unikraft run --metro fra \
  -m 256M \
  -p 443:8080/http+tls \
  --image <my-org>/nginx:latest

This single command will (a) create a service via the -p flag and (b) start an NGINX instance:

Code
metro:        fra
name:         nginx-67zbu
uuid:         8a8bc1b9-0af6-420e-a426-190dc2da9eaa
state:        starting
image:        <my-org>/nginx
resources:
  memory:     256MiB
  vcpus:      1
service:
  uuid:       a942b9b5-ad17-3ffe-dcd2-ef4331f9087a
  name:       nameless-fog-0tvh1uov
  domains:
  - fqdn:     nameless-fog-0tvh1uov.fra.unikraft.app
networks:
- uuid:       62d9bbf0-aec8-61f6-7bdb-86edf63dd068
  private-ip: 10.0.3.3
  mac:        12:b0:c6:23:ed:15
timestamps:
  created:    just now

With this in place, it's now time to start a second instance and attach it to the created service (in this case, named nameless-fog-0tvh1uov):

Code
unikraft run --metro fra \
  -m 256M \
  --service nameless-fog-0tvh1uov \
  --image <my-org>/nginx:latest

You should see another instance starting up, with the same service and domain as the first instance:

Code
metro:        fra
name:         nginx-0s295
uuid:         9b1c8e5c-7a0d-4f1e-9c3a-2b8d9f0e5a1f
state:        starting
image:        <my-org>/nginx
resources:
  memory:     256MiB
  vcpus:      1
service:
  uuid:       a942b9b5-ad17-3ffe-dcd2-ef4331f9087a
  name:       nameless-fog-0tvh1uov
  domains:
  - fqdn:     nameless-fog-0tvh1uov.fra.unikraft.app
networks:
- uuid:       adf1f17d-bad1-49b2-a126-ed730e06379a
  private-ip: 10.0.1.4
  mac:        12:b0:0a:00:01:05
timestamps:
  created:    just now

To check that it worked, run the following command:

Code
unikraft services get nameless-fog-0tvh1uov

You should see output such as:

Code
metro:         fra
name:          nameless-fog-0tvh1uov
uuid:          a942b9b5-ad17-3ffe-dcd2-ef4331f9087a
persistent:    true
limits:
  soft:        1
  hard:        65535
timestamps:
  created:     just now
domains:
- fqdn:        nameless-fog-0tvh1uov.fra.unikraft.app
instances:
- name:        nginx-0s295
  uuid:        9b1c8e5c-7a0d-4f1e-9c3a-2b8d9f0e5a1f
- name:        nginx-67zbu
  uuid:        8a8bc1b9-0af6-420e-a426-190dc2da9eaa
services:
- source:      443
  destination: 8080
  handlers:    ["tls", "http"]

Note the two instances under the instances field. You're now load balancing across 2 NGINX instances!

Code
unikraft instance stop --drain-timeout 5s my-instance

Load balancing algorithm

The load balancing algorithm is a variant of least_conn. For every instance, the system tracks the number of current in-flight TCP connections (if in tcp mode) or requests (if in http mode).

To select an instance, the system goes over all instances in the service. It finds the set of instances with the least amount of in-flight requests/connections, and picks from that set.

To illustrate, imagine this scenario:

In this case, the algorithm first chooses instances i-1 and i-3, since they both have the least number of connections at the moment (only 1 each). After that, it chooses between these 2 instances and assigns the new connection to one of them. For example, if it chose i-1, the next new connection goes to i-3. This is because i-3 now becomes the only instance with only 1 connection (assuming none of the connections that the other instances handle close).

Learn more

• The CLI reference.
• Unikraft Cloud's REST API reference, and in particular the section on services.
• Instances documentation.
• Service groups documentation.