3

I'm trying to understand how various factors affect LLMs. Specifically the size of the dataset they're trained on.

What would be the main difference between:

  • A regular LLM (like ChatGPT) that's trained on the entire internet
  • Same LLM but trained on a very small dataset, like just one book - harry potter

Would it still be as proficient at language, if not the knowledge?

Example: If I posed the question "How long did the COVID pandemic last?", would it still try to answer in perfect English but without the actual information, like "Ah, COVID, that pesky little poltergeist that's been plaguing the Muggle world for longer than a troll under the Whomping Willow!"

Or will it just be gibberish because one book is not enough for it to learn the complexity required to formulate a response in English?

How small can the dataset get till it just becomes a really fancy fuzzy search?

Example: "What's harry's last name" "Potter Harry Stone Rowling"

1
  • Two good answers below, so I will just add a comment. The miracle of the transformer architecture is attention. The attention mechanism in its full form relies on multiple different embeddings for words and even phrases. Without adequate examples, this mechanism will not be able to judge context well. The classic 'do you mean apple the juice or Apple the company' question. That is why in your example, it would never map COVID to poltergeist.
    – JP Alioto
    Commented Jan 30 at 17:48

2 Answers 2

2

One book? Not a chance. To take Project Gutenberg's edition of Moby Dick as an example, there are a total of 212,057 words in the book; there are 18,660 distinct words (including hyphenated words).

Defining a "context" as a distinct sequence of three words, 8901 of those words are used in just a single context, so an LLM will have no chance of figuring out how to use a word like "sack", "dozen", or "uneven" from an analysis of Moby Dick. Only 2115 words are used in ten or more contexts, making it extremely difficult to understand words like "usual", "larger", or "blanket". Just 231 words are used in a hundred or more contexts, and 26 words ("the", "of", "and", "a", "to", "in", "that", "his", "it", "I", "but", "he", "as", "with", "was", "for", "all", "this", "at", "by", "not", "from", "him", "so", "on", "be") are used in more than a thousand contexts.

An LLM might be able to string together simple grammatical sentences about whales after an analysis of Moby Dick, but I expect the actual result would be no better than a simple Markov chain generator.

1

Some ideas in addition to Mark's answer:

  1. Training a typical LLM on a very small dataset will likely overfit.
  2. One can use https://github.com/karpathy/nanoGPT/ to train on a small dataset to see how the generated will look like. They provide some ready-to-use training code on some Shakespeare text.
  3. The quality of the training set may help reduce the training set size without reducing performances too much, e.g. see this paper from Microsoft Research: Textbooks Are All You Need.
  4. Inverse Scaling: When Bigger Isn’t Better

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.