DeepSpeed

Model Setup

• Training Setup
  • Argument Parsing
  • Training Initialization
  • Distributed Initialization
• Inference Setup
  • DeepSpeedInferenceConfig
  • DeepSpeedTPConfig
  • DeepSpeedMoEConfig
  • QuantizationConfig
  • InferenceCheckpointConfig

Training API

• Training API
  • Forward Propagation
  • Backward Propagation
  • Optimizer Step
  • Gradient Accumulation
  • Model Saving

Inference API

• Inference API
  • Forward Propagation

Checkpointing API

• Model Checkpointing
  • Loading Training Checkpoints
  • Saving Training Checkpoints
  • ZeRO Checkpoint fp32 Weights Recovery
  • Avoiding ZeRO Checkpoint Bloat
  • Universal Checkpoints (under development)
• Activation Checkpointing
  • Configuring Activation Checkpointing
  • Using Activation Checkpointing
  • Configuring and Checkpointing Random Seeds

ZeRO API

• ZeRO
  • Getting Started
  • Constructing Massive Models
  • Manual Parameter Coordination
  • Memory-Centric Tiling
  • Debugging
  • Modifying Partitioned States
  • GPU Memory Management
  • Offload States

Mixture of Experts (MoE)

• Mixture of Experts (MoE)
  • Layer specification

Transformer Kernel API

• Transformer Kernels
  • DeepSpeed Transformer Config
  • DeepSpeed Transformer Layer

Pipeline Parallelism

• Pipeline Parallelism
  • Model Specification
  • Training
  • Extending Pipeline Parallelism

Optimizers

• Optimizers
  • Adam (CPU)
  • FusedAdam (GPU)
  • FusedLamb (GPU)
  • OneBitAdam (GPU)
  • ZeroOneAdam (GPU)
  • OnebitLamb (GPU)

Learning Rate Schedulers

• Learning Rate Schedulers
  • LRRangeTest
  • OneCycle
  • WarmupLR
  • WarmupDecayLR
  • WarmupCosineLR

Flops Profiler

• Flops Profiler
  • FlopsProfiler
  • get_model_profile()

Autotuning

• Autotuning
  • Autotuner

Memory Usage

• Memory Requirements
  • API To Estimate Memory Usage
  • Discussion

Monitoring

• Monitoring
  • TensorBoard
  • WandB
  • Comet
  • CSV Monitor

Indices and tables

• Index

• Module Index

• Search Page