DeepSpeed

Model Setup

• Training Setup
  • Argument Parsing
  • Training Initialization
  • Distributed Initialization
• Inference Setup

Training API

• Training API
  • Forward Propagation
  • Backward Propagation
  • Optimizer Step
  • Gradient Accumulation
  • Model Saving

Inference API

• Inference API
  • Forward Propagation

Checkpointing API

• Model Checkpointing
  • Loading Training Checkpoints
  • Saving Training Checkpoints
  • ZeRO Checkpoint fp32 Weights Recovery
• Activation Checkpointing
  • Configuring Activation Checkpointing
  • Using Activation Checkpointing
  • Configuring and Checkpointing Random Seeds

ZeRO API

• ZeRO
  • Getting Started
  • Constructing Massive Models
  • Manual Parameter Coordination
  • Memory-Centric Tiling

Mixture of Experts (MoE)

• Mixture of Experts (MoE)
  • Layer specification
  • Groups initialization

Transformer Kernel API

• Transformer Kernels
  • DeepSpeed Transformer Config
  • DeepSpeed Transformer Layer

Pipeline Parallelism

• Pipeline Parallelism
  • Model Specification
  • Training
  • Extending Pipeline Parallelism

Optimizers

• Optimizers
  • Adam (CPU)
  • FusedAdam (GPU)
  • FusedLamb (GPU)
  • OneBitAdam (GPU)
  • OnebitLamb (GPU)

Learning Rate Schedulers

• Learning Rate Schedulers
  • LRRangeTest
  • OneCycle
  • WarmupLR
  • WarmupDecayLR

Flops Profiler

• Flops Profiler

Autotuning

• Autotuning
  • Autotuner

Memory Usage

• Memory Requirements
  • API To Estimate Memory Usage
  • Discussion

Indices and tables

• Index
• Module Index
• Search Page