DeepSpeed

Model Setup

• Training Setup
  • Argument Parsing
  • Training Initialization
  • Distributed Initialization
• Inference Setup
  • DeepSpeedInferenceConfig
  • DeepSpeedTPConfig
  • DeepSpeedMoEConfig
  • QuantizationConfig
  • InferenceCheckpointConfig

Training API

• Training API
  • Forward Propagation
  • Backward Propagation
  • Optimizer Step
  • Gradient Accumulation
  • Mixed Precision Training
  • Configuring ZeRO Leaf Modules
  • Model Saving
  • Training Multiple Models
  • Automatic Tensor Parallel Training
  • Automatic Sequence Parallel Training

Inference API

• Inference API
  • Forward Propagation
  • HybridEngine Rollout Profiling
  • Shared Prompt Prefill
  • Continuous batching (experimental)

Checkpointing API

• Model Checkpointing
  • Loading Training Checkpoints
  • Saving Training Checkpoints
  • ZeRO Checkpoint fp32 Weights Recovery
  • Avoiding ZeRO Checkpoint Bloat
  • Universal Checkpoints (under development)
• Activation Checkpointing
  • Configuring Activation Checkpointing
  • Using Activation Checkpointing
  • Configuring and Checkpointing Random Seeds

ZeRO API

• ZeRO
  • Getting Started
  • Constructing Massive Models
  • Manual Parameter Coordination
  • Memory-Centric Tiling
  • Debugging
  • Modifying Partitioned States
  • GPU Memory Management
  • Offload States

Mixture of Experts

• AutoEP (Automatic Expert Parallelism)
  • Experimental regional torch.compile
• Mixture of Experts (DeepSpeed MoE)
  • MoE

Transformer Kernel API

• Transformer Kernels
  • DeepSpeed Transformer Config
  • DeepSpeed Transformer Layer

Pipeline Parallelism

• Pipeline Parallelism
  • Model Specification
  • Training
  • Extending Pipeline Parallelism

Optimizers

• Optimizers
  • Adam (CPU)
  • FusedAdam (GPU)
  • FusedLamb (GPU)

Learning Rate Schedulers

• Learning Rate Schedulers
  • LRRangeTest
  • OneCycle
  • WarmupLR
  • WarmupDecayLR
  • WarmupCosineLR

Flops Profiler

• Flops Profiler
  • FlopsProfiler
  • get_model_profile()

Autotuning

• Autotuning
  • Autotuner

Memory Usage

• Memory Requirements
  • API To Estimate Memory Usage
  • Discussion
• Host Memory Pinning
  • Pin on/off vs backend
  • Backend Selection
  • Native device registration
  • Requirements for native
  • Lifetime and unpinning (native)

Monitoring

• Monitoring
  • TensorBoard
  • WandB
  • Comet
  • CSV Monitor

Indices and tables

• Index

• Module Index

• Search Page