Zero-Reload Model Switching with vLLM Sleep Mode

Introduction

The multi-model serving problem: You have two LLMs that each fit on your GPU, but not both at once. Traditional solutions force a bad tradeoff:

1. Keep both models loaded → Requires 2x the GPU memory (expensive, often impossible)
2. Reload models on-demand → 30-100+ seconds per switch (slow, wasteful)

vLLM Sleep Mode offers a third way: Models hibernate in seconds and wake up fast—delivering the efficiency of on-demand loading with the speed of persistent serving.

Two Sleep Levels for Different Needs

• Level 1: Offloads weights to CPU RAM (fast wake time)
• Level 2: Discards weights entirely (nearly as fast wake time, minimal RAM usage)

Both levels are 18-200x faster than full reload and work seamlessly with Tensor Parallelism (TP), Pipeline Parallelism (PP), and Expert Parallelism (EP).

Why Sleep Mode Beats Fast Weight Loaders

Even with instant weight loading, every cold start pays hidden costs that Sleep Mode avoids:

By keeping the process alive, Sleep Mode preserves infrastructure (#2-4) and avoids expensive reinitialization. This is why benchmarks show Sleep Mode inference is 61-88% faster than cold starts.

This post covers:

• Comprehensive benchmarks across model sizes (0.6B to 235B) and GPUs (A4000 to A100)
• Technical deep-dives explaining the performance gains
• Ablation studies on warm-up impact and FP8 quantization
• Decision guide for choosing the right sleep level

Quick Start: Using Sleep Mode

Online Serving API

Start two vLLM servers with Sleep Mode enabled:

# Terminal 1: Start Phi-3-vision
export VLLM_SERVER_DEV_MODE=1
vllm serve microsoft/Phi-3-vision-128k-instruct --enable-sleep-mode --port 8001

# Terminal 2: Start Qwen3-0.6B
export VLLM_SERVER_DEV_MODE=1
vllm serve Qwen/Qwen3-0.6B --enable-sleep-mode --port 8002

Sleep and Wake Models

# Put Phi-3-vision to sleep (Level 2 - minimal RAM usage)
curl -X POST 'localhost:8001/sleep?level=2'

# Put Qwen3-0.6B to sleep (Level 2)
curl -X POST 'localhost:8002/sleep?level=2'

# Wake up Phi-3-vision for inference
curl -X POST 'localhost:8001/wake_up'
curl -X POST 'localhost:8001/collective_rpc' \
  -H 'Content-Type: application/json' \
  -d '{"method":"reload_weights"}'

# IMPORTANT: Reset prefix cache after waking (Level 2 only)
curl -X POST 'localhost:8001/reset_prefix_cache'

# Now run inference on Phi-3-vision...
# (your inference requests here)

# Put back to sleep when done
curl -X POST 'localhost:8001/sleep?level=2'

# Wake up Qwen3-0.6B
curl -X POST 'localhost:8002/wake_up'
# (Level 1 doesn't need reload_weights or reset_prefix_cache)

# Run inference on Qwen3-0.6B...

Performance Overview

Let’s see how Sleep Mode performs compared to traditional model reloading.

Sleep Mode L1 vs No Sleep Mode Performance

The interactive chart below shows the total time to perform 5 model switches: running inference on Model A, switching to Model B, running inference on Model B, then repeating this pattern (A→B→A→B→A→B).

With Sleep Mode: Models sleep/wake between switches, preserving infrastructure. Without Sleep Mode: Each switch requires a full vLLM restart and reload.

Inference Performance Boost

Beyond faster model switching, Sleep Mode also delivers faster inference times. Because models are already warmed up when woken from sleep, they skip the cold start overhead that affects freshly loaded models.

Why Sleep Mode Improves Inference Speed

The 61-88% inference speedup isn’t from faster weight loading—it’s from preserving expensive infrastructure that cold starts must rebuild from scratch.

What Sleep Mode Preserves:

The Critical Difference:

• Without Sleep Mode: Process dies on unload → You CANNOT benefit from pre-warm-up
  • Must restart Python process and CUDA context
  • Must reinitialize memory allocator
  • Must re-capture CUDA graphs
  • Must re-JIT compile kernels (DeepGEMM, FlashInfer, TorchInductor)
  • Result: First inference is 4-7x slower (see benchmarks: 0.92s wake vs 3.72s cold start)
• With Sleep Mode: Process stays alive → Pre-warm-up pays off
  • ✅ Allocator, graphs, process state, and JIT kernels all preserved after initial warmup
  • Result: First inference stays fast (~1s), avoiding the 3-4s cold start penalty

Model Switching Performance

The most dramatic benefit of Sleep Mode is in model switching time. Waking a sleeping model is 18-20x faster than loading a fresh vLLM instance.

Hardware Scalability: A4000 GPU Results

Sleep Mode benefits aren’t limited to high-end GPUs. Here’s the same workload on an A4000 GPU with smaller models, demonstrating that the performance gains scale across different hardware tiers and model sizes.

A4000: Inference Performance

A4000: Model Switching Performance

Key Observations on A4000:

• Inference Performance: Wake mode delivers 83% faster inference for Qwen3-0.6B and 81% faster for Phi-3-vision
• Model Switching: Wake times are incredibly fast (~0.1-0.8s), achieving 58-203x speedup vs cold starts
• Total time savings: 62% (85s vs 226s for 5 model switches)
• Near-instant switching for small models (0.1s wake time), making multi-model serving feel seamless
• Demonstrates that Sleep Mode is effective across different GPU classes and model sizes

Sleep Levels: Choosing the Right Mode

vLLM Sleep Mode offers two levels with different tradeoffs:

Level 1 (Default): Offloads model weights to CPU memory, discards KV cache

• Fastest wake times (~0.1-0.8s for small models, ~3-6s for large models)
• Requires sufficient CPU RAM to store model weights
• Best for: Systems with adequate CPU memory, frequent model switching

Level 2: Discards model weights and KV cache, keeps only buffers (rope scaling tensors, etc.) in CPU

• Slower wake times (~0.8-2.6s for small models) due to weight reload from disk
• Minimal CPU RAM usage - only small buffers retained
• Best for: Systems with limited CPU RAM or when managing many models that won’t all fit in memory

Performance Comparison: Level 1 vs Level 2 vs No Sleep

Performance Summary:

Key Insights:

• Level 1 is fastest (68% faster than no sleep) but needs significant CPU RAM
• Level 2 is nearly as fast (65% faster than no sleep) with minimal RAM requirements
• Level 2 wake is ~3x slower than Level 1 (0.85s vs 0.26s for Qwen3-0.6B) due to weight reload
• Both sleep modes deliver massive improvements over no sleep mode

Why Level 2 is Still Faster Than No Sleep Mode

At first glance, this seems counterintuitive: Level 2 reloads weights from SSD (just like “No Sleep Mode”), so why is it 23-45x faster overall?

The Answer: Weight loading is only ONE of FIVE costs

When you reload a model without Sleep Mode, you pay all these costs:

Level 2 Strategy:

• Weight reload from SSD (same as No Sleep)
• Everything else preserved: Process state, allocator instance, CUDA graphs, and compiled JIT kernels all intact
• No recompilation needed: Kernels were compiled during initial warmup and remain cached
• Average per switch: ~2.6s (see benchmark data above)

No Sleep Mode Reality:

• Weight reload from SSD (same as Level 2)
• Everything else rebuilt: Process restart + allocator init + graph re-capture
• JIT kernels: Full compilation + explicit warm-up routine (kernel_warmup() + dummy runs)
• Average per switch: ~48s (see benchmark data above)

The benchmark data proves it: For 5 model switches:

• Level 2: 124.6s total (average ~2.6s per switch)
• No Sleep: 357.1s total (average ~48s per switch)

Even though both reload weights from SSD, Level 2 is 2.9x faster overall because it preserves the expensive infrastructure (process state, allocator, CUDA graphs) that No Sleep Mode must rebuild from scratch every single time.

Level 2: Inference Performance

Level 2: Model Switching Performance

Key Observations:

When to Use Level 2:

• Limited CPU RAM: System cannot hold all model weights in CPU memory
• Cost Optimization: Cheaper cloud instances with less CPU RAM
• Many Models: Switching between many models where CPU memory is a constraint
• Still Significant Gains: Even with weight reload, Level 2 is 23-45x faster than no sleep mode

Level 1 vs Level 2 Comparison:

• Level 1: ~0.1-0.8s wake time, needs ~10-100GB+ CPU RAM per model
• Level 2: ~0.8-2.6s wake time, needs only ~MB CPU RAM per model
• Both dramatically faster than full reload (~20-100s)

Ablation Studies

Impact of Warm-Up on Sleep Mode

Does skipping the warm-up phase affect performance? Warm-up pre-compiles CUDA graphs during initial load, which can take several seconds. Let’s compare with and without warm-up.

Key Findings:

Insights:

• Warm-Up Compiles Kernels Once, Benefits All Wake Cycles: With initial warmup, JIT compilation and CUDA graph capture happen once during load and are preserved across all subsequent sleep/wake cycles
• Without Warm-Up, Every Wake-Up Pays Compilation Cost: The 5-7x slowdown happens on the first inference after every single wake-up, not just once
• Compiled Kernels Are Preserved Across Sleep/Wake: After warmup during initial load (8.4s), all subsequent wake-ups have fast first inference (0.45s, 0.93s) proving kernels stay cached
• Minimal Warmup Sufficient: A single 1-token inference is enough to trigger full JIT compilation and CUDA graph capture, making warmup very cheap
• Trade Initial Load Time for Consistent Performance: The 8.4s warmup cost is paid once and amortized across all model switches
• Recommendation: Always Use Warm-Up for production workloads where consistent, fast inference is expected

Impact of Quantization on Sleep Mode

Does quantization (FP8) affect Sleep Mode performance? We tested the same workload with and without FP8 quantization on A100 GPU.

Ablation: Inference Performance (BF16 vs FP8)

Ablation: Model Switching (BF16 vs FP8)

Key Findings:

Insights:

• FP8 has faster wake operations (13-33% faster) due to less memory movement
• FP8 improves inference for larger models (30% faster for Phi-3-vision) but shows minimal difference for tiny models
• Initial load takes longer with FP8 due to quantization overhead during warmup
• After initial load, FP8 provides smoother switching with faster wake cycles
• For workloads with frequent switching, FP8’s faster wake times can offset the longer initial load

Decision Guide: Which Sleep Level to Use?

Use Sleep Level 1 When:

• You have sufficient CPU RAM to hold all model weights
• You need the fastest possible wake times (0.1-6s)
• You’re switching models very frequently (every few seconds/minutes)
• Inference latency consistency is critical

Use Sleep Level 2 When:

• CPU RAM is limited (can’t hold all model weights)
• You’re optimizing cloud costs (cheaper instances with less RAM)
• You have many models to manage (10+)

Skip Sleep Mode When:

• You’re only using a single model (no switching needed)
• Model switches are extremely rare (once per day/week)
• Both models fit simultaneously in GPU memory

Conclusion

vLLM Sleep Mode transforms multi-model GPU serving from a 30-100 second reload penalty into sub-second switches. The benchmarks speak for themselves:

• 18-200x faster model switching depending on model size and hardware
• 61-88% faster inference for warmed models vs cold starts
• 65-68% total time savings across complete workloads
• Works at every scale: 0.6B to 235B parameters, small and large GPUs

The future of LLM serving is multi-model. Sleep Mode makes it practical today.

Acknowledgements

Special thanks to Vensen Mu, Jeff Aw, Jun Kang Chow, Tun Jian Tan, Pin Siang Tan, Amir Balwel, Ye Hur Cheong, Zhiyao Cen and Kaichao You for developing the Sleep Mode feature and this blog post.