2

Here is what I observed for fine tuning Flan-T5-base:

GPU: 1 Nvidia V100 with 16 GB memory.
Flan-T5-base model size: 990 MB.
Script: deep-learning-pytorch-huggingface.
Samsum train dataset size: 370 MB.
Batch size: 8.
==> This will use 16 GB GPU memory(obtained from nvidia-smi command) and take about 1.5 hr to finish training.

Chnage to batch_size = 1.
==> This will use 7 GB GPU memory and take about 7.5 hr to finish training.

I would like to know if there is a rule of thumb which can be used to estimate GPU memory size and training time for fine tuning a LLM given GPU type and model, train data and batch sizes.

1 Answer 1

1

I found an article regarding this. According to it: "The equation ties together the throughput 𝜏, the training time T, the model size N and the number of training tokens T: 𝜏T = 6ND "

However, this is more of an estimate assuming peak throughput and other factors, so you should probably refer to the article for some context.

Here is the link.

Edit: Formatting

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.