--- title: "Say No to Idle GPUs" description: "The industry has spent three years buying GPUs and almost no time using them well. Up to 70% of a fleet can sit idle because a card stays pinned to an instance that is doing nothing. Here is why an idle GPU should belong to someone else โ€” and how GPU snapshots make that possible." doc_version: "1.0" last_updated: "2026-10-05" --- The industry has spent three years buying GPUs and almost no time using them well, and that trend is set to continue for the time being. We built something that radically changes that: a GPU now belongs to a VM only while it runs. Here is why that matters more than the next procurement order of H100s. GPUs are the most expensive thing in a data center, and often times, at least for workloads like inference, rendering and transcoding, they are doing nothing. Not slowly. Not inefficiently. Nothing. They sit attached to a VM that is waiting for its next request, its next tool call, its next user, and nobody else is allowed to touch them. We think that is the single largest inefficiency in AI infrastructure today. We also think it is fixable, and we have built the fix. As AI companies continue the GPU gold rush, where the name of the game is grab as many as you can as fast as you can, we strongly believe it is high time that we take a serious look at improving how efficiently they are operated. ## The gold....errr...GPU rush Talk to anyone running GPUs at scale and the conversation is about procurement. How many cards, which generation, how soon. Capacity is the metric; utilization rarely comes up. It is a rational response to scarcity. During the COVID pandemic, for instance, when supply was uncertain, people bought 500 rolls of toilet paper 100 packets of pasta in one go, leaving the rest of us wondering why the aisles were empty. The uncertainty was the driver, and, with GPUs, the same human instinct is at play: we don't know what the supply will look like, and we all have a strong feeling that the demand will be essentially infinite, at least in the short to middle term. And yes, I've just done a GPU-to-toilet-paper analogy -- so now you know that this wasn't AI generated and no GPUs were involved nor harmed in the process ๐Ÿ™‚. Going into our own research, we guessed GPUs sat idle about 30% of the time. Then we spoke with teams building GPU observability tooling, the people who actually watch the counters, and one founder put the number closer to 70%. Time for a back of the envelop calculation then as to the order of magnitude cost of running fleets of GPUs inefficiently: Clearly you could plug all kinds of different numbers into the calculation, but this just gives a feeling for it. Why doesn't anybody fix it? We could make several hypotheses: Likely the landscape is composed of a combination of these three. ## Why idle GPUs stay idle The waste is not a scheduling bug. It is structural. On almost every platform, a GPU is bound to an instance for as long as that instance exists, running or not. And the workloads we put on GPUs are exactly the ones that are busy in bursts. - **Inference.** Waits between requests. Traffic is spiky; the card is provisioned for the peak. - **Agents.** Wait on tool calls, APIs and humans. Seconds of compute, minutes of holding. - **Small models.** Need a fraction of a card and get the whole thing anyway. The usual answer is a warm pool: keep instances loaded so the next request is fast. Warm pools trade latency for waste, and with GPUs the waste is priced in thousands of dollars per card. Scale-to-zero, the obvious alternative, has never worked for GPUs because a cold GPU workload has to reload the model, rebuild its state and warm its caches. Even with the container image already cached, that is tens of seconds for a small model and minutes for a large one, against a time-to-first-token expectation that is normally well under a second. ## A GPU is assigned only while its instance runs With our new GPU snapshotting support, when a GPU instance suspends or scales to zero, the GPU's state is saved with the snapshot and the GPU is released. Whoever needs it next gets it. When the instance resumes, it picks up exactly where it left off, on the same GPU if it is free or on another of the same model. Suspend and resume take about one to two seconds, state included (that number may go up somewhat if the GPU model/slice has larger memory capacity). The important detail is that we do not snapshot the GPU on its own. CPU and GPU state are captured together, in one snapshot of the whole VM: memory, registers, device state and GPU memory, frozen at the same instant. That is what makes the restore coherent. > In essence, this mechanism takes GPU inefficiency from the long, cumulative periods of GPU idleness to a model where it only consists of the brief moments taken to suspend and resume VMs. Which means GPU workloads can finally scale to zero like everything else on the platform. Idle inference endpoints and headful browsers suspend, stop consuming node resources, and resume on the next request. **Available now ยท Enterprise:** GPU snapshots, NVIDIA vGPU, and scale-to-zero for GPU and browser workloads are available today on Unikraft Enterprise. [Read the docs โ†’](/docs) ## What comes next: GPU workloads that move Scale-to-zero is the first thing GPU snapshots unlock. It is not the last. A coherent snapshot of a running GPU workload is portable state, and portable state can be moved. That lays the groundwork for GPU workload migration: suspend on one node, resume on another, or on a different cluster entirely. Once GPU workloads can move, the fleet stops being a set of fixed assignments and becomes a pool. Workloads can be packed onto fewer nodes so the rest power down. Hot spots can be drained before they become outages. Nodes can be patched and serviced without killing long-running sessions. Capacity in one cluster can absorb demand from another, and work can follow cheaper power or spare cards across regions. Time-sharing gets more work out of each GPU. Migration gets more work out of the whole fleet. ## What we believe 1. **An idle GPU belongs to someone else.** If it is not computing, it should be free for whoever is. 2. **State is not a reason to hold hardware.** Snapshot it, release the card, resume in seconds. 3. **Warm pools are a tax, not a strategy.** When resume is fast enough, keeping things warm is just paying for nothing. 4. **Right-size the card.** Small models get a slice, not a server. 5. **Measure utilization, not capacity.** The number of GPUs you own matters less than the number doing work. ## Efficiency is the next land grab Buying GPUs was phase one. It will not last as a strategy. Supply will loosen, budgets will tighten, and the question will shift from "how many cards do you have" to "how much work do they do." The teams that answer it well will serve the same demand with half the hardware, or twice the demand with the same. We think a 2โ€“3ร— improvement in GPU utilization is sitting in most fleets today, waiting for the infrastructure to stop pinning cards to instances that are not using them. GPU snapshots are our first step toward realizing that potential. ## Sitemap - [Home](https://unikraft.com/) - [How It Works](https://unikraft.com/how-it-works) - [AI](https://unikraft.com/ai) - [Pricing](https://unikraft.com/pricing) - [Customer Stories](https://unikraft.com/customer-stories) - [Blog](https://unikraft.com/blog) - [Documentation](https://unikraft.com/docs) - [Glossary](https://unikraft.com/glossary.md) - [Contact](https://unikraft.com/contact)