Index

A

allgather_bucket_size (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) allgather_partitions (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

allgather_sequential (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) api_key (deepspeed.monitor.config.CometConfig attribute)

B

BackwardPass (class in deepspeed.runtime.pipe.schedule) buffer_count (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute) (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadParamConfig attribute)

buffer_size (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadParamConfig attribute) BufferOpInstruction (class in deepspeed.runtime.pipe.schedule)

C

clone_tensors_for_torch_save() (in module deepspeed.checkpoint.utils) CommitP2P (class in deepspeed.runtime.pipe.schedule) compute_grad_norm (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) contiguous_gradients (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) convert_zero_checkpoint_to_fp32_state_dict() (in module deepspeed.utils.zero_to_fp32)

convert_zero_checkpoint_to_state_dict() (in module deepspeed.utils.zero_to_fp32) cpu_offload (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) cpu_offload_param (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) cpu_offload_use_pin_memory (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) cpuadam_cores_perc (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute)

D

DataParallelSchedule (class in deepspeed.runtime.pipe.schedule) deepspeed.runtime.pipe.schedule module

DeepSpeedCPUAdam (class in deepspeed.ops.adam) device (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute) (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadParamConfig attribute) DualPipeVSchedule (class in deepspeed.runtime.pipe.schedule)

E

elastic_checkpoint (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) enable_sanity_checks (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) enabled (deepspeed.monitor.config.CometConfig attribute) (deepspeed.monitor.config.CSVConfig attribute) (deepspeed.monitor.config.TensorBoardConfig attribute) (deepspeed.monitor.config.WandbConfig attribute)

experiment_key (deepspeed.monitor.config.CometConfig attribute) experiment_name (deepspeed.monitor.config.CometConfig attribute)

F

fast_init (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute) filter_match() (deepspeed.runtime.pipe.ProcessTopology method) forward() (deepspeed.moe.layer.MoE method)

ForwardPass (class in deepspeed.runtime.pipe.schedule) FusedAdam (class in deepspeed.ops.adam) FusedLamb (class in deepspeed.ops.lamb)

G

gather_16bit_weights_on_model_save (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) get_axis_comm_lists() (deepspeed.runtime.pipe.ProcessTopology method) get_axis_list() (deepspeed.runtime.pipe.ProcessTopology method) get_axis_names() (deepspeed.runtime.pipe.ProcessTopology method) get_coord() (deepspeed.runtime.pipe.ProcessTopology method)

get_default_autocast_lower_precision_modules() (in module deepspeed.runtime.torch_autocast) get_dim() (deepspeed.runtime.pipe.ProcessTopology method) get_fp32_state_dict_from_zero_checkpoint() (in module deepspeed.utils.zero_to_fp32) get_rank() (deepspeed.runtime.pipe.ProcessTopology method) get_rank_repr() (deepspeed.runtime.pipe.ProcessTopology method) group (deepspeed.monitor.config.WandbConfig attribute)

I

ignore_unused_parameters (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) InferenceSchedule (class in deepspeed.runtime.pipe.schedule) init_autocast_params() (in module deepspeed.runtime.torch_autocast)

is_autocast_initialized() (in module deepspeed.runtime.torch_autocast) is_first_stage (deepspeed.runtime.pipe.schedule.PipeSchedule property) is_last_stage (deepspeed.runtime.pipe.schedule.PipeSchedule property)

J

job_name (deepspeed.monitor.config.CSVConfig attribute) (deepspeed.monitor.config.TensorBoardConfig attribute)

L

leaf_module (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) legacy_stage1 (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) load_from_fp32_weights (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

load_state_dict_from_zero_checkpoint() (in module deepspeed.utils.zero_to_fp32) LoadMicroBatch (class in deepspeed.runtime.pipe.schedule) log_trace_cache_warnings (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) LRRangeTest (class in deepspeed.runtime.lr_schedules)

M

max_in_cpu (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadParamConfig attribute) max_live_parameters (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) max_reuse_distance (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) memory_efficient_linear (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) mode (deepspeed.monitor.config.CometConfig attribute)

model_persistence_threshold (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) module deepspeed.runtime.pipe.schedule module_granularity_threshold (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) MoE (class in deepspeed.moe.layer)

N

num_micro_batches (deepspeed.runtime.pipe.schedule.PipeSchedule property) num_pipe_buffers() (deepspeed.runtime.pipe.schedule.DataParallelSchedule method) (deepspeed.runtime.pipe.schedule.DualPipeVSchedule method) (deepspeed.runtime.pipe.schedule.InferenceSchedule method) (deepspeed.runtime.pipe.schedule.PipeSchedule method) (deepspeed.runtime.pipe.schedule.TrainSchedule method)

num_stages (deepspeed.runtime.pipe.schedule.PipeSchedule property) nvme_path (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute) (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadParamConfig attribute)

O

offload_optimizer (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) offload_param (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) OneCycle (class in deepspeed.runtime.lr_schedules) online (deepspeed.monitor.config.CometConfig attribute)

OptimizerStep (class in deepspeed.runtime.pipe.schedule) output_path (deepspeed.monitor.config.CSVConfig attribute) (deepspeed.monitor.config.TensorBoardConfig attribute) overlap_comm (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) override_module_apply (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

P

param_persistence_threshold (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) parameter_alignment (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) pin_memory (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute) (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadParamConfig attribute) PipeInstruction (class in deepspeed.runtime.pipe.schedule) pipeline_loading_checkpoint (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

pipeline_read (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute) pipeline_write (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute) PipeSchedule (class in deepspeed.runtime.pipe.schedule) prefetch_bucket_size (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) ProcessTopology (class in deepspeed.runtime.pipe) project (deepspeed.monitor.config.CometConfig attribute) (deepspeed.monitor.config.WandbConfig attribute)

R

ratio (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute) RecvActivation (class in deepspeed.runtime.pipe.schedule) RecvGrad (class in deepspeed.runtime.pipe.schedule) reduce_bucket_size (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

reduce_scatter (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) ReduceGrads (class in deepspeed.runtime.pipe.schedule) ReduceTiedGrads (class in deepspeed.runtime.pipe.schedule) round_robin_gradients (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

S

safe_get_full_fp32_param() (in module deepspeed.utils) safe_get_full_grad() (in module deepspeed.utils) safe_get_full_optimizer_state() (in module deepspeed.utils) safe_get_local_fp32_param() (in module deepspeed.utils) safe_get_local_grad() (in module deepspeed.utils) safe_get_local_optimizer_state() (in module deepspeed.utils) safe_set_full_fp32_param() (in module deepspeed.utils) safe_set_full_grad() (in module deepspeed.utils) safe_set_full_optimizer_state() (in module deepspeed.utils) safe_set_local_fp32_param() (in module deepspeed.utils) safe_set_local_grad() (in module deepspeed.utils) safe_set_local_optimizer_state() (in module deepspeed.utils)

safe_update_full_grad_vectorized() (in module deepspeed.utils) samples_log_interval (deepspeed.monitor.config.CometConfig attribute) save_muon_momentum_buffer_in_memory (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) SendActivation (class in deepspeed.runtime.pipe.schedule) SendGrad (class in deepspeed.runtime.pipe.schedule) stage (deepspeed.runtime.pipe.schedule.PipeSchedule property) (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) stage3_gather_fp16_weights_on_model_save (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) steps() (deepspeed.runtime.pipe.schedule.DualPipeVSchedule method) (deepspeed.runtime.pipe.schedule.PipeSchedule method) sub_group_size (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) super_offload (deepspeed.runtime.zero.config.DeepSpeedZeroOffloadOptimizerConfig attribute)

T

team (deepspeed.monitor.config.WandbConfig attribute)

TrainSchedule (class in deepspeed.runtime.pipe.schedule)

U

use_all_reduce_for_fetch_params (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

use_multi_rank_bucket_allreduce (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

W

WarmupCosineLR (class in deepspeed.runtime.lr_schedules) WarmupDecayLR (class in deepspeed.runtime.lr_schedules)

WarmupLR (class in deepspeed.runtime.lr_schedules) workspace (deepspeed.monitor.config.CometConfig attribute)

Z

zenflow (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) zero_hpz_partition_size (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)

zero_quantized_gradients (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) zero_quantized_nontrainable_weights (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute) zero_quantized_weights (deepspeed.runtime.zero.config.DeepSpeedZeroConfig attribute)