2

Based on this answer, the LLaMA 3 model is more dense than GPT-4. Is the number of tokens per parameter dependent on the training data? Is there a way to increase it during training/development?

Improve this question

1 Answer 1

Reset to default
2

Is the number of tokens per parameter dependent on the training data?

Yes. Number of tokens per parameter = number of tokens in training set / number of parameters in the trained model.

Is there a way to increase it during training/development?

Scrape more data. Which is why some greedy corporations try to paywall access to data, e.g. see https://meta.stackexchange.com/q/401324/178179, in an attempt to milk AI companies and researchers, thereby slowing down AI progress.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.