Will quantization lower model quality?

8-bit quantization is typically close to lossless; 4-bit approaches like AWQ or GPTQ may cause small drops on certain tasks. Always verify on your own evaluation set.

What does continuous batching mean?

It is a serving technique that slots new requests into the GPU batch as others complete, rather than waiting for an entire batch to finish, which raises throughput considerably.

Which serving engine is best for LLMs?

vLLM is a solid default for open-weight models; SGLang and TensorRT-LLM can be quicker in particular setups. Benchmark using your own traffic.

Can a self-hosted setup keep an OpenAI-compatible API?

Yes. Most current serving engines provide OpenAI-compatible endpoints, so applications can change models with very little code change.

Cutting LLM Inference Cost Without Hurting Quality

Lower LLM inference cost per token with quantization, continuous batching, prefix caching, speculative decoding and right-sized GPUs.

What you pay for LLM inference comes down to GPU time per token. The largest savings typically come from a modern serving engine with continuous batching, 8-bit or 4-bit quantization, prefix caching for repeated prompts, the smallest model that clears your quality bar, and GPUs that autoscale to actual traffic. Measure cost per million tokens and latency percentiles before and after each change.

What determines the cost of LLM inference?

Each token generated needs a pass through the model, and GPUs cost money whether working or idle. Cost therefore hinges on model size, how many requests share each GPU concurrently, the length of prompts and outputs, and how much idle capacity you are paying for.

Users perceive latency in two parts: time to first token, mostly spent processing the prompt, and time between tokens, mostly governed by generation speed and contention.

The optimisation levers with the biggest impact

Roughly ordered by typical impact for self-hosted models:

Picking the right GPU for LLM serving

Select by memory first: model weights plus the KV cache for your concurrency and context length have to fit. Bigger GPUs like the H100 or H200 suit large models and high throughput; L40S and L4 class cards are cost-effective for smaller or quantized models.

Combine capacity types: reserved capacity for steady baseline load, and on-demand or spot capacity for peaks and batch jobs. Scale-to-zero helps internal tools that sit idle at times.

Self-hosting versus an LLM API

Managed APIs are the simplest option and frequently the cheapest at low or bursty volume. Self-hosting open-weight models gets attractive when volume is high and steady, when data has to remain inside your perimeter, or when a fine-tuned small model can handle the job.

Plenty of teams run both behind an AI gateway that sends each request to the cheapest model meeting its quality requirement, falling back when a provider is slow or down.

Measuring inference improvements

Benchmark using realistic traffic — your real prompt lengths, output lengths and concurrency — rather than synthetic single requests. Track throughput (tokens per second per GPU), time to first token, P95 latency, GPU utilisation and cost per million tokens.

Re-run your quality evaluation after each optimisation. Quantization and smaller models only count as savings if answers remain correct.

A practical plan for optimising inference

Begin by measuring, then adjust one lever at a time:

Right-size the model: a smaller model that is well prompted or fine-tuned frequently equals a larger one on a specific task.
Adopt a serving engine with continuous batching and paged attention — vLLM, SGLang or TensorRT-LLM — so many requests share the GPU efficiently.
Quantize weights (AWQ, GPTQ or FP8) to shrink memory and raise throughput with minimal quality loss — confirm this on your own evaluation set.
Turn on prefix and KV-cache reuse when many requests use the same system prompt or documents.
Experiment with speculative decoding, in which a small draft model suggests tokens for the large model to verify.
Trim prompts: leaner system prompts and tighter retrieved context cut the work done on every request.
Profile today's traffic and define cost and latency targets.
Switch to a modern serving engine and turn on batching and caching.
Test quantized and smaller models against your evaluation set.
Right-size the GPUs and set up autoscaling.
Introduce a gateway for routing, caching and cost dashboards.
Our AI services cover LLM inference and GPU infrastructure work from start to finish.