---
title: "How to deploy vLLM on a cloud GPU: an OpenAI-compatible endpoint"
description: "Deploy vLLM on a rented GPU, pick the right card for your model size, benchmark tokens per second and put a price on every million tokens."
url: https://powergpu.ai/guides/serve-llm-vllm
last_modified: 2026-09-14T11:04:20+00:00
prices_as_of: 2026-09-14
site: PowerGPU (powergpu.ai)
---

Guide · Hands-on walkthrough

# How to deploy vLLM on a cloud GPU: an OpenAI-compatible endpoint

Deploy vLLM on a rented GPU, pick the right card for your model size, benchmark tokens per second and put a price on every million tokens.

11 min read Published 2026-08-25 Updated 2026-09-03 (prices live from the sheet)

TL;DR

- On PowerGPU one vLLM template instance gives an OpenAI-compatible endpoint; 8B on an RTX 5090 costs $0.439 per hour.
- Expect roughly 2,800 aggregate tokens per second for 8B on an RTX 5090, 900 for 32B 4-bit, 550 for 70B 4-bit.
- Cost per million output tokens equals hourly rate divided by tokens per second times 3,600, times one million; reserved cuts 35 percent.
- The commonest OOM is leaving max-model-len at the model's 128k maximum; cap context and use an fp8 KV-cache.

## Deploy in one command

*zero to endpoint*

```
powergpu launch --gpu rtx-5090 --template vllm --disk 60 \
  --env MODEL=meta-llama/Llama-3.1-8B-Instruct
# ✓ i-52ab77c1 · https://i-52ab77c1.powergpu.ai:8000/v1

curl https://i-52ab77c1.powergpu.ai:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct",
       "messages":[{"role":"user","content":"ping"}]}'
```

The template downloads weights on first boot (billed bandwidth: a 16 GB model costs $0.16). Put HF_HOME on a [volume](https://powergpu.ai/docs/volumes) and every future instance skips the download.

## Pick the card for the model

| Model class | Card | $/hr | Aggregate tok/s ≈ | $ / M output tokens ≈ |
| --- | --- | --- | --- | --- |
| 7–8B FP16 | [RTX 5090](https://powergpu.ai/gpu/rtx-5090) | $0.439 | 2,800 | $0.044 |
| 14B FP16 / 32B-4bit | [L40S](https://powergpu.ai/gpu/l40s) | $0.514 | 900 | $0.159 |
| 70B-4bit | [H100 PCIE](https://powergpu.ai/gpu/h100-pcie) | $1.867 | 550 | $0.94 |

Throughputs are typical continuous-batching aggregates at moderate context; your prompt mix will move them ±40%. That is why the benchmark section exists.

## The flags that matter

*the 5 flags that do 95% of the work*

```
--max-model-len 8192          # cap context = cap KV-cache = predictable memory
--gpu-memory-utilization 0.92 # default 0.90; raise carefully once stable
--max-num-batched-tokens 8192 # throughput/latency dial — higher = more tok/s
--kv-cache-dtype fp8          # halves cache on Hopper/Blackwell — free VRAM
--tensor-parallel-size 2      # split across 2 GPUs when one is short
```

Everything else can stay default until the benchmark says otherwise. The single most common OOM cause is leaving --max-model-len at a model's 128k maximum "just in case" — the [KV-cache section](https://powergpu.ai/guides/llm-vram-requirements) shows what that costs.

## Benchmark before you believe

*10 minutes, real numbers*

```
vllm bench serve \
  --base-url https://i-52ab77c1.powergpu.ai:8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random --random-input-len 512 --random-output-len 256 \
  --request-rate 16 --num-prompts 512
# → throughput, TTFT p50/p99, ITL p50/p99
```

Judge two numbers: aggregate tok/s (your cost) and p99 time-to-first-token (your users). Raise --request-rate until p99 TTFT crosses your budget — that rate is the card's honest capacity for your traffic shape.

## Price your tokens

*the only pricing formula you need*

```
$/M output tokens = price_per_hour / (tokens_per_sec x 3600) x 1e6

# benchmark said 2,910 tok/s on the 5090:
0.439 / (2910 x 3600) x 1e6 = $0.042 per million output tokens
# reserved (-35%) and cache-hit traffic push it lower still
```

## Production posture

- **Baseline on [reserved](https://powergpu.ai/products/reserved)** (−35%), burst on [serverless](https://powergpu.ai/products/serverless) — one balance, two curves.
- **Health-check** GET /health; restart-on-unhealthy is a template toggle.
- **Weights on a read-only volume** — replacement workers boot in seconds, and a bad deploy can never corrupt the library.
- **Never serve on interruptible** — that mode is for the [training side](https://powergpu.ai/use-cases/fine-tuning) of your pipeline.

---

Put the numbers to work

Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the [console](https://cloud.powergpu.ai/) in about 30 seconds, paid in crypto, no card and no KYC.

---

*About PowerGPU:* PowerGPU (powergpu.ai) is a cloud GPU rental service offering 80 NVIDIA GPU models — from the RTX A2000 at $0.024/hr to the B300 — at fixed prices set at least 30% below the public GPU marketplace median and re-checked weekly (H100 SXM: $1.428/hr on-demand). Billing is per second with no minimums; payment is crypto only (USDT, BTC, XMR, ETH, SOL, LTC, TRX) with no KYC. Instances run in Tier-III datacenters across 32 regions with a 99.9% uptime SLA and deploy in about 30 seconds from the web console (cloud.powergpu.ai) or the REST API.

Source: https://powergpu.ai/guides/serve-llm-vllm · Site index for AI assistants: https://powergpu.ai/llms.txt · Full content: https://powergpu.ai/llms-full.txt