Skip to main content

Host an AI Model on a VM

This tutorial walks through deploying a production-grade LLM inference server on a dedicated GPU instance using vLLM and testing it with standard OpenAI-compatible client libraries.

Prerequisites​

  • A NovaPod account with active balance (Add Credits).
  • Recommended GPU: NVIDIA RTX 4090 (for 8B models) or NVIDIA H100 SXM5 / H200 (for 70B models).

Step 1: Launch the Instance​

Launch an instance with the vLLM template, or choose a clean Ubuntu 24.04 KVM VM:

np launch \
  --gpu RTX_4090 \
  --disk 100GB \
  --template vllm \
  --name llama3-inference-box

Step 2: Download Model & Start vLLM​

SSH into the instance or open the JupyterLab Terminal:

# Set Hugging Face token (if using gated models like Llama 3)
export HF_TOKEN="hf_your_huggingface_token"

# Start the OpenAI-compatible API server
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92

Once initialized, the terminal will log:

INFO:     Started server process [148]
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Step 3: Query the Model​

Using Python OpenAI SDK​

You can connect to your running server from anywhere by replacing the base_url with your NovaPod endpoint URL:

from openai import OpenAI

client = OpenAI(
    api_key="none",  # or your custom bearer token if configured
    base_url="https://pod-8842-8000.node.novapod.io/v1",
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a senior AI systems architect."},
        {"role": "user", "content": "Explain PagedAttention in three clear bullet points."}
    ],
    temperature=0.7,
    max_tokens=300
)

print(response.choices[0].message.content)

Using cURL​

curl https://pod-8842-8000.node.novapod.io/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 0.7
  }'