Inference API

deepspeed.init_inference() returns an inference engine of type InferenceEngine.

for step, batch in enumerate(data_loader):
    #forward() method
    loss = engine(batch)

Forward Propagation

deepspeed.InferenceEngine.forward(self, *inputs, **kwargs)

Execute forward propagation

Parameters:

• *inputs – Variable length input list
• **kwargs – variable length keyword arguments