1

As I know, it will take several months to pre-train an LLM like chatGPT. I am wondering, what is the industry standard practice to do a pilot run to make sure that everything is correct (as it will be too costly to figure out that something is wrong after finish training the entire model) for an LLM?

1 Answer 1

0

how to do a quick pilot run when pre-training a large language model?

  1. Train a much smaller LLM (e.g., 100M or 1B parameters).
  2. Monitor the evolution of the losses as the model is training.
1
  • that makes sense! Thank you!
    – Dmitry J
    Commented Oct 31, 2023 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.