GPU Inference on a Budget: Optimizing Costs for Self-Hosted Models

Once your AI application reaches sufficient scale, the math on self-hosted model inference often starts to beat managed API pricing. A GPT-4 call costs roughly $30/million tokens. Running an equivalent open-source model — Llama 3.1 70B, Qwen 2.5, or Mistral — on your own GPU infrastructure can bring that figure down to $2-5/million tokens at scale.

But “self-hosted” is deceptively simple-sounding. Getting the infrastructure right requires real expertise. Here’s what we’ve learned building GPU inference stacks for production.

Choose the Right Serving Framework

The serving framework you choose has a massive impact on both throughput and cost efficiency.

vLLM is our current default for most production deployments. Its PagedAttention mechanism dramatically improves GPU memory utilization, enabling far higher throughput per GPU than naive implementations. It supports continuous batching, streaming, and most major model architectures.

Triton Inference Server is excellent when you need tight NVIDIA ecosystem integration, multi-model serving on a single GPU, or advanced ensemble pipelines.

llama.cpp and Ollama are great for local development and CPU inference but aren’t production serving frameworks — don’t try to use them at scale.

Quantization: The Best Free Performance Gain

Quantization reduces model weight precision (from FP32/FP16 to INT8 or INT4), shrinking model size and improving inference speed — often with surprisingly little quality degradation.

We typically start with AWQ (Activation-aware Weight Quantization) or GPTQ quantization to INT4. For a 70B model, this brings GPU memory requirements from ~140GB to ~35GB, enabling it to run on a single A100/H100 80GB GPU instead of requiring 2-4 GPUs.

Always measure quality impact with your specific evaluation suite before deploying quantized models to production.

Autoscaling and Spot Instances

Don’t run always-on GPU instances for variable workloads. Use autoscaling groups with GPU spot/preemptible instances for the bulk of your capacity, with a small always-on baseline to handle the minimum load.

On AWS, G5 or P4d instances with spot pricing can give you 60-70% cost savings vs on-demand. On GCP, A2 preemptible instances are similarly cost-effective.

Speculative Decoding for Latency

If your primary constraint is latency rather than throughput, speculative decoding is often worth the engineering investment. A smaller “draft” model generates token candidates that the larger “target” model then verifies in parallel. For many tasks, this yields 2-3x latency improvements with identical output quality.

The catch: it requires careful tuning of the draft model and acceptance criteria for your specific use case.

---

Self-hosted inference is a significant infrastructure investment, but it’s often the right move at scale. If you’re exploring whether it makes sense for your use case, we’re happy to run the numbers with you.