GPU spend is the line item that surprises ML teams most, in both directions: projects stall waiting for hardware that was never needed, and cloud bills arrive for GPUs that sat idle at 15% utilization. This post is a practical budgeting guide for 2026: how to think about cloud versus owned hardware, how to use spot capacity without losing work, what a representative fine-tune actually consumes, and when bringing in outside help is cheaper than learning by burning compute.
We deliberately avoid quoting dollar prices. They change monthly and differ by region, provider and commitment. Every provider publishes a current price list; the arithmetic below is what to do with it.
Start with utilization, not capacity
Before buying or renting anything, measure what you have. On every GPU you pay for, log utilization (nvidia-smi dmon or DCGM exported to your metrics stack) for two weeks. In our experience most teams find their training GPUs under 40% utilized over a week (idle nights, idle weekends, data-loading bottlenecks, Python-bound training loops) and their inference GPUs under 30%. Fixing the software is almost always cheaper than buying hardware: BF16, torch.compile, a fused optimizer and a proper data loader often double effective capacity for free. Do that first and re-measure.
Cloud versus owned hardware
The crossover is about utilization and time horizon.
Rent (on-demand or reserved) when: workloads are bursty or experimental, you need the newest GPU generation, your team is small, or you cannot commit to a 2-3 year horizon. Reserved or committed-use pricing typically cuts on-demand rates substantially for a one- or three-year commitment; it is the right move for a steady inference fleet you have already sized.
Own when: you can keep GPUs above roughly 60-70% utilization around the clock for two years or more, you have (or can buy) the operational capability to run them, and data-residency rules make cloud awkward. A well-utilized owned GPU is generally cheaper over three years than the equivalent on-demand rental, but an owned GPU at 30% utilization is the most expensive option there is.
Colocated or bare-metal GPU providers sit in between and have become a serious option for training clusters: reserved physical GPUs with no hypervisor overhead, often cheaper than hyperscaler on-demand, with less platform tooling.
Run the arithmetic per workload, not per company: a steady inference fleet and a bursty research queue usually want different answers.
Spot strategies that actually work
Spot and preemptible capacity is typically discounted heavily against on-demand, and it is how cost-conscious teams train. The rules:
- Checkpoint frequently with distributed checkpointing. A preemption should cost the minutes since the last save, not the run. Every 15-30 minutes is a reasonable default for long jobs; see Distributed training with FSDP2.
- Make jobs resumable by default. Training code should find the latest checkpoint and continue without a human.
- Diversify instance types and zones. Ask for any of several equivalent GPU types so a shortage in one pool does not stall you.
- Keep the orchestrator on on-demand capacity. The thing that restarts preempted workers must not itself be preemptible.
- Do not run latency-critical inference on spot unless you have enough on-demand capacity to absorb a full spot reclamation.
What a 7B fine-tune actually consumes
Teams routinely overestimate this. For a QLoRA fine-tune of a 7-8B model on 10,000 instruction examples at sequence length 1024, two epochs:
- Hardware: one 24 GB GPU is sufficient; one 80 GB GPU is comfortable and roughly twice as fast.
- Time: on the order of one to four GPU-hours depending on the GPU and the sequence lengths.
- Storage: the base model (about 16 GB in BF16), adapters (tens of MB), merged weights (about 16 GB) and checkpoints.
That is a small fraction of a single day's on-demand GPU cost at current prices. A full fine-tune (all parameters, BF16, FSDP2 across 8 GPUs) on the same data is perhaps 10-20 GPU-hours. The expensive part of a fine-tuning project is never the training compute; it is data preparation, evaluation and the engineers' time. Budget accordingly: for most projects, compute is under 10% of the cost.
Inference is where the money goes. A fine-tuned 8B model served in INT4 or FP8 on vLLM handles thousands of tokens per second on one GPU; the question is how many such GPUs your traffic needs at peak and what fraction of the day they sit idle. Model it as cost per million tokens for each candidate configuration and compare against API pricing honestly; at low volumes an API is cheaper, at high volumes or with privacy constraints self-hosting wins. See Serving an LLM with vLLM.
Budget lines teams forget
- Evaluation compute. Running a 2,000-example evaluation on every candidate model adds up; it is the best money you will spend.
- Storage and egress. Checkpoints accumulate; egress on a multi-terabyte dataset moved between providers can exceed the training cost.
- Idle development GPUs. Notebook instances left running overnight are a classic leak. Auto-stop them.
- The retry tax. Runs that fail at hour 20 because of an OOM or a NaN that could have been caught at minute 2. Smoke-test every run on a tiny subset first.
When a consultancy saves money
Bringing in outside help is cheaper than compute in three situations: when a team is about to buy or commit to hardware without a utilization baseline; when a training or serving workload has never been profiled (the first profiling pass typically halves the cost); and when a project is burning GPU-weeks on runs that a regression harness and a smaller experiment would have avoided. In each case a short engagement pays for itself in the first month of the bill it prevents.
If you would like a cost model for a specific workload, our AI consulting and strategy and LLM inference optimization practices build exactly that. Contact us with your current GPU bill and we will tell you where it is going.