Skip to main content

Persistent Environments

When developing on interactive GPU pods, reinstalling packages (e.g., custom CUDA wheels, flash-attn, unsloth, specific transformers branches) on every launch wastes valuable compute credits and developer time.

NovaPod provides two mechanisms to ensure your development environments survive pause, restart, and hardware re-allocation.

1. Virtual Environments in /workspace (Docker Pods)​

On all containerized Docker GPU Pods, the /workspace directory is backed by your allocated persistent NVMe volume.

Creating a Persistent Virtualenv​

# Navigate to persistent workspace
cd /workspace

# Create virtual environment
python3 -m venv my_env

# Activate and install dependencies
source my_env/bin/activate
pip install --upgrade pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
pip install vllm flash-attn

Auto-Activating on Shell Login​

Add the activation script to /workspace/.bashrc_custom:

echo "source /workspace/my_env/bin/activate" >> /workspace/.bashrc_custom

When your pod is paused and resumed later, activating the virtualenv takes 0 seconds.

2. Full OS State Persistence (Root KVM VMs)​

If you use a Root KVM VM, you do not need to restrict yourself to /workspace.

  • Any package installed via apt-get install, system services created under /etc/systemd/system/, custom Docker images cached in /var/lib/docker, and global python packages installed via pip install stay 100% frozen on your persistent system drive.
  • When the VM resumes, your system is in the exact byte-for-byte state it was when paused.

3. Startup Scripts​

You can configure an automated Startup Script in your NovaPod dashboard or CLI to run automatically every time a new instance boots:

#!/usr/bin/env bash
# Automatically executed on container startup
pip install wandb bitsandbytes
git clone https://github.com/your-org/model-repo.git /workspace/repo