Self-Hosted LLM Inference on Dedicated GPUs
Financial Services & Banking — Customer data was not allowed outside the firm's own cloud account, and per-token API bills were rising faster than usage.…
Client: Mid-sized Financial Services Company. Sector: Financial Services & Banking.
The challenge: Customer data was not allowed outside the firm's own cloud account, and per-token API bills were rising faster than usage.
What we built: Ran open-weight LLMs within the client's cloud on autoscaling GPU nodes, using quantised vLLM serving, an AI gateway that handles routing, fallbacks and caching, and complete latency and cost telemetry.
Tech used: vLLM, Kubernetes, NVIDIA GPUs, AI Gateway, Prometheus.
- Cost of Inference: -60%
- P95 Response Time: <800 ms
- Data Leaving the Cloud: Zero
- Availability: 99.9%