Skip to main content

Serverless LLM Endpoints

NovaPod Serverless allows developers to deploy open-weight foundation models (Llama 3, DeepSeek, Qwen 2.5, Mistral, Gemma) as fully managed, auto-scaling HTTPS endpoints. You bring the model; we manage the GPU allocation, serving engine, request queueing, and scale-to-zero lifecycle.

Key Capabilities​

  • Zero Idle Costs (Scale-to-Zero): When no requests arrive, worker nodes spin down automatically. You pay $0 for compute while idle.
  • OpenAI Drop-In Compatibility: Use your existing OpenAI Python / TypeScript SDK code without changing parameter formats.
  • Powered by vLLM & SGLang: State-of-the-art PagedAttention, continuous batching, and chunked prefill engines delivering up to 3x higher token throughput than standard Hugging Face TGI.
  • Global DDoS & HTTPS Termination: Automatic TLS certificate management, custom domains, and rate-limiting guards.

Deploying an Endpoint​

Via CLI​

np deploy create \
  --name deepseek-coder-v2 \
  --model deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct \
  --engine vllm \
  --gpu H100_SXM \
  --min-workers 0 \
  --max-workers 4 \
  --idle-timeout 300

Once provisioned, you will receive a permanent HTTPS endpoint:

https://api.novapod.io/v1/endpoints/deepseek-coder-v2

Querying Your Endpoint​

from openai import OpenAI

client = OpenAI(
    api_key="np_live_your_secret_key",
    base_url="https://api.novapod.io/v1/endpoints/deepseek-coder-v2/v1"
)

stream = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct",
    messages=[{"role": "user", "content": "Write a high-performance CUDA kernel for matrix multiplication."}],
    stream=True
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Pricing Model​

  • Active Worker Billing: Billed strictly per minute of active GPU worker execution.
  • Cold-Start Buffer: Pre-cached weights in local datacenter caches reduce cold-boot latency to under 20 seconds.