2

What is the total number of web pages used to train ChatGPT-3.5, and what methods are employed to ensure the veracity of the vast data while filtering out inaccurate information?

2
  • Please take a look to genai.meta.stackexchange.com/a/122/12.
    – Wicket
    Jul 31, 2023 at 1:37
  • 1
    I don't have enough rep to initiate the close vote for this question, but I think this question needs more focus. As it is currently written, this post is asking 2 different questions at the same time.
    – holydragon
    Jul 31, 2023 at 3:06

2 Answers 2

3

What is the total number of web pages used to train ChatGPT-3.5

This information hasn't been released. For comparison, here's the GPT-3 training data size, from https://arxiv.org/pdf/2005.14165.pdf:

enter image description here

3

"Number of web pages" isn't really a meaningful measure. The usual measure is "number of tokens", where a token is, on average, about three-quarters of a word. GPT-3 was trained on roughly 430 billion tokens of web content (plus data from other sources); detailed training information for ChatGPT-3.5 has not been released.

The usual method for ensuring veracity in large datasets is to assume that accurate information is repeated, while erroneous information is unique. This results in accurate information being reinforced during training when it's encountered many times. However, in practice, widespread errors and well-known falsehoods also get reinforced. (Alpaca, for example, gives a well-known but incorrect explanation when asked about the origin of the word "posh".)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.