Skip to main content

Troubleshooting Guide

1. CUDA Out of Memory (OOM) Errors​

Symptom: RuntimeError: CUDA out of memory. Tried to allocate X GiB

Solutions:

  • Enable Gradient Checkpointing: Reduces memory consumption by recomputing activations during backprop: 
    model.gradient_checkpointing_enable()
  • Use 4-bit / 8-bit Quantization: Use bitsandbytes or unsloth to load model weights in FP8 or NF4: 
    from transformers import BitsAndBytesConfig
    bnb_config = BitsAndBytesConfig(load_in_4bit=True)
  • Reduce Micro-Batch Size: Decrease per_device_train_batch_size and compensate by increasing gradient_accumulation_steps.
  • vLLM GPU Memory Utilization: If serving with vLLM, ensure --gpu-memory-utilization is tuned between 0.90 and 0.95.

2. SSH Connection Refused or Timeout​

Symptom: ssh: connect to host node-42.novapod.io port 22042: Connection refused

Solutions:

  • Wait for Cold Boot: Give newly launched instances 15 to 30 seconds to finish initializing the SSH daemon and injecting your public key.
  • Verify Public Key: Ensure the public key you uploaded in Console Settings matches the private key on your local machine (ssh -i ~/.ssh/id_ed25519 -p 22042 root@...).
  • Check Security Groups: If attached to a custom security group, verify that port 22 is open to your current public IP address.

3. Disconnected JupyterLab Kernel​

Symptom: Kernel dies immediately upon loading a large model.

Solution: The Linux Out-Of-Memory (OOM) killer likely terminated the Python process due to system RAM exhaustion (not GPU VRAM). When allocating an instance, ensure system host RAM is at least 1.5x the size of the uncompressed model weights.