DeepSpeed

Model Setup

Training API

Inference API

Checkpointing API

ZeRO API

Mixture of Experts (MoE)

Transformer Kernel API

Pipeline Parallelism

Optimizers

Learning Rate Schedulers

Flops Profiler

Autotuning

Memory Usage

Monitoring

Indices and tables