Supported devices

Accelerator labels

We believe that the types of accelerators used to build AI data centers will continue to diversify, and the MoAI Inference Framework provides features to efficiently utilize clusters with heterogeneous accelerators.

Each accelerator is identified by labels prefixed with moai.moreh.io. Users can either specify the accelerator on which a model should run, or allow the framework to automatically select the optimal accelerator. Below is the list of accelerator labels supported by the MoAI Inference Framework.

Checking available accelerators

You can use the following command to see which accelerators are currently available in the cluster and how they are labeled by the MoAI Inference Framework.

kubectl get nodes -L moai.moreh.io/accelerator.vendor,moai.moreh.io/accelerator.model

NAME         STATUS     ROLES           AGE    VERSION   ACCELERATOR.VENDOR   ACCELERATOR.MODEL
cpu-001      Ready      control-plane   1d     v1.32.5
mi250-001    Ready      <none>          1d     v1.32.5   amd                  mi250
mi300x-001   Ready      <none>          1d     v1.32.5   amd                  mi300x

To list nodes that match a specific label, use the -l option:

kubectl get nodes -l moai.moreh.io/accelerator.model=mi300x

NAME         STATUS   ROLES    AGE   VERSION
mi300x-001   Ready    <none>   1d   v1.32.5