# 2. Preparing for Fine-tuning

For a smooth tutorial experience, the following specifications are recommended:

• CPU: 16 cores or more

• Memory: 512GB or more

• MAF version: 24.5.0

• Storage: 500GB or more

Please verify that your environment meets these requirements before starting the tutorial.

To start, you'll need to obtain a container or virtual machine on the MoAI Platform from your infrastructure provider. You can use public cloud services based on the MoAI Platform, such as:

• MoAI Platform Trial Container (Inquiries: support@moreh.io)

• KT Cloud Hyperscale AI Computing

After accessing the platform via SSH, run the moreh-smi command to ensure the MoAI Accelerator is properly recognized. Note that device names may vary depending on the system.

For this tutorial, which involves training a large-scale language model (LLM) like Llama3, selecting the appropriate size of MoAI Accelerator is crucial. First, use the moreh-smi command to check the current MoAI Accelerator in use.

Details on the specific MoAI Accelerator settings required for training will be provided in 3. Model Fine-tuning

$ moreh-smi
+---------------------------------------------------------------------------------------------------+
|                                                  Current Version: 24.5.0  Latest Version: 24.5.0  |
+---------------------------------------------------------------------------------------------------+
|  Device  |        Name         |      Model     |  Memory Usage  |  Total Memory  |  Utilization  |
+===================================================================================================+
|  * 0     |   MoAI Accelerator  |  xLarge.512GB  |  -             |  -             |  -            |
+---------------------------------------------------------------------------------------------------+

Setting up the PyTorch script execution environment on the MoAI Platform is similar to working on a standard GPU server.

Once you’ve accessed the container via SSH, check if PyTorch is installed in the current conda environment by running:

$ conda list torch
...
# Name                    Version                   Build  Channel
torch                     1.13.1+cu116.moreh24.5.0          pypi_0    pypi
...

The version should display both the PyTorch version and the MoAI version it’s running on. For instance, 1.13.1+cu116 indicates PyTorch version 1.13.1 with CUDA 11.6, and MoAI version 24.5.0.

If you see a conda: command not found message, the torch package isn’t listed, or the torch package doesn’t include "moreh" in its version name, follow the instructions in the Prepare Fine-tuning on MoAI Platform document to create the conda environment.

Run the following to ensure the torch package is properly imported and that the MoAI Accelerator is recognized:

$ python
>>> import torch
...
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name()
[info] Requesting resources for MoAI Accelerator from the server...
[info] Initializing the worker daemon for MoAI Accelerator
[info] [1/1] Connecting to resources on the server (192.168.110.00:24158)...
[info] Establishing links to the resources...
[info] MoAI Accelerator is ready to use.
'MoAI Accelerator'
>>> quit()

Download the PyTorch script for training from the GitHub repository by running:

For this tutorial, we will use the train_llama3.py script located in the tutorial directory.

$ sudo apt-get install git
$ git clone https://github.com/moreh-dev/quickstart.git
$ cd quickstart
~/quickstart$ ls tutorial
...  train_llama3.py  ...

Install third-party Python packages needed to run the script by executing:

$ pip install -r requirements/requirements_llama3.txt

To access and download the Llama3 70B model checkpoint from Hugging Face Hub, you will need to agree to the community license and provide your Hugging Face token information.

First, enter the necessary information and agree to the license on the Hugging Face website.

Once you've submitted the agreement form, check that the status on the page has updated as follows:

Now you can authenticate your Hugging Face token with the following command:

huggingface-cli login