---
title: "The Idle (and Over-provisioned) Cloud"
description: "We generally like to think of the cloud handling gazillion requests per second, but a significant part of it is over-provisioned and so idle. In this post we look into the what, how and why what we term \"The Idle Cloud\" exists, and how to fix it."
doc_version: "1.0"
last_updated: "2024-09-11"
---

Let’s start with the elephant in the room: while the cloud has certainly
revolutionized the way we do business, and handles unimaginable amounts of
traffic per day, it is massively over-provisioned and so also often idle. There
are many reports and accounts attesting to this, but let’s take Datadog's recent
[State of Cloud Costs report](https://www.datadoghq.com/state-of-cloud-costs/)
which nicely quantifies the problem:

<div class="w-full max-w-xl mx-auto mt-4">
  <a href="https://www.datadoghq.com/state-of-cloud-costs/" class="group mt-6 block border border-slate-300 dark:border-slate-600 p-4 hover:border-blue-500 hover:-translate-y-1 transition-all transition-100 transform-gpu dark:hover:border-slate-200">
    
  </a>
  <div class="italic text-slate-500 dark:text-slate-400 px-0 py-2 text-center">
    <span>Idle resources account for a majority of container costs.</span>
  </div>
</div>

So "idle resources account for a majority of container costs" – their words, not
mine.  And the problem isn't limited to "just" containers; reports along these
lines are common all across cloud services:

<object data="/images/charts/cost-of-the-cloud-articles.svg" type="image/svg+xml" class="w-full">
  <img src="/images/charts/cost-of-the-cloud-articles.svg" alt="Cost of the Cloud Articles" />
</object>

So we have what we term "The Idle Cloud", but where is all of this idling coming
from? Some of it of course comes from people allocating resources, especially
for dev purposes, rarely or never using them, and then forgetting to turn them
off.  There are many tools to help identify such cases and to shut down unused
resources.

But this isn't where the big problem lies, because much of the idleness comes
from production systems that are over-provisioned to cope with peak traffic, or
to ensure that services respond with minimal delay. We do this because traffic
(and traffic bursts) happen in timescales of milliseconds, but the cloud was
never built for such short timescales.

<div class="border border-slate-300 dark:border-blue-100 p-8 pb-0">
  <h3 class="text-2xl pt-0 prose-p:my-0 my-0">
      The fundamental reason behind the Idle Cloud is the orders of magnitude
      dissonance between millisecond demands from traffic being serviced (or
      attempted to be serviced) by cloud services that take seconds or even
      minutes to come up.
  </h3>
  
</div>

More concretely, this dissonance manifests itself in the form of a large range
of common, constantly recurring cloud infra issues, which are commonly solved by
throwing money at the problem, aka, over-provisioning:

| Issue                      | Current (Over-provisioning) Workaround |
|----------------------------|----------------------------------------|
| Extremely slow cold starts | Keep instances warm by artificially pinging them, or pay more to “boost” cold start times. |
| Slow scale to zero         | Keep instances on longer (minutes) than they need to. |
| Slow auto-scale            | Provision for peak capacity always or use a FaaS platform but then deal with a number of other shortcomings (eg, run duration limits, limited language support, functions only, etc). |
| Low server density         | Keep idle instances always on and use many more servers than needed. |

Many people even resort to renting bare metal servers even for intermittent
workloads (the ultimate form of over-provisioning: practically owning a server
and running it forever) because that's cheaper than the workarounds above. The
cloud is not only idle but also messy – a topic we've explored at length
elsewhere in our writing on the cloud engineer's dilemma.

## Cloud traffic has Millisecond Semantics. Shouldn't Cloud Platforms Too?

Asking cloud infra designed to operate in seconds or minutes to service traffic
that is intermittent or bursty in millisecond timescales is bound to cause
problems. At Unikraft, we believe it's time to disrupt the current way we build
cloud infra, as well as the technology we use to build it with, as well as the
technology we use to build cloud infra, to one where it can be reactive, in
milliseconds timescales. We call this **true serverless**, because we feel the
promise of serverless was that devs would just deploy apps and services and
would never have to even think about infra issues (the “less” in serverless) –
but that promise has unfortunately been largely broken: devs are very much aware
that there’s servers/cloud infra underneath, and, worse, there’s little they can
control or do about fixing the root problem (nor should they!) other than
through over-provisioning. One could even argue that the incentives for cloud
infra providers to fix these issues are mis-aligned, in that they make more
money out of these inefficiencies, but let’s table that for another day.

Our Unikraft Cloud platform provides true, millisecond semantics: cold starts,
autoscale, and scale to zero are all extremely snappy, all of the while
providing strong, hardware-level isolation via virtual machines (the golden
standard for multi-tenancy on public clouds) and the ability to deploy complex
services based on `Dockerfile`s, not just functions.  As an example, and to
illustrate, here’s what happens with scale to zero on current cloud infra (top)
and on Unikraft Cloud (bottom):

Another important result of these millisecond semantics is that we can have
massive server density: imagine you have a large user base consisting of 1
million users, but a long tail of inactive users. With normal infra, you’d have
to provision for 1 million users, but with a millisecond, reactive platform, you
could just provision for the max number of concurrently active users. In many
cases, the difference between that number and the number of total users can be
several orders of magnitude, and so with Unikraft Cloud this means you could
service your 1M users with a fraction of the servers you’d normally need. In
some stress tests, we’ve had as many as 100,000 instances running on a single,
standard server.  I won't cover how we achieve milliseconds semantics in this
post; for a high level explanation of this you can check out this [how it
works](/how-it-works) page.

## Sitemap

- [Home](https://unikraft.com/)
- [How It Works](https://unikraft.com/how-it-works)
- [AI](https://unikraft.com/ai)
- [Pricing](https://unikraft.com/pricing)
- [Customer Stories](https://unikraft.com/customer-stories)
- [Blog](https://unikraft.com/blog)
- [Documentation](https://unikraft.com/docs)
- [Glossary](https://unikraft.com/glossary.md)
- [Contact](https://unikraft.com/contact)