What works better: An LLM trained on better texts or an LLM with better prompts?

Question

Assume you have a very large corpus of high-quality documents related to a given topic, and assume you have a pretrained large language model (the foundation model) with training data not containing these documents.

Scenario 1: You use the high-quality documents as additional pretraining data, possibly with a large weight.

Scenario 2: You create a vector database for the high-quality documents, i.e. you split them up in chunks and calculate text embeddings for them.

Now assume you ask the language model a question on the topic. In scenario 1 the question just gets answered (probably better than by the original foundation model), in scenario 2 the text embedding of the question prompt is calculated and the prompt then enhanced by the most similar text chunks in the vector database.

Does anyone have an intuition (and could explain it) which scenario would yield better results?

Answer 1

Scenario 3: provide your high-quality documents to your user, via a good search engine that shows excerpts. Since these documents are high-quality for an LLM to use as training data, everything in them is true when taken out of context: so just do that. (Do you want to risk the near-certainty of the model saying wrong stuff in the same authoritative style as your source texts?) Plenty of existing software solutions will do this for you, at significantly lower expense than running an LLM. Cēterum cēnsō dēlendam esse GPT quā vēritātem.

If you truly have to decide between scenarios 1 and 2, it depends how similar the texts are to the model. If the texts are similar enough, Scenario 1 effectively lossy-compresses their contents, making it the obvious choice; however, if the texts are different enough, embeddings won't be a faithful representation and you need to modify the model (à la Scenario 2) to make it more inclined to produce those texts. A better option would be to do both of these, since that gets you closest to the behaviour of Scenario 3.

Cēterum autem cēnsō dēlendam esse GPT quā vēritātem. Use tools only where they are suitable.

Answer 2

I think in this scenario, the results of using "model + vector database" will be better. Because from your background description, it seems that your questions are all in the dataset. As long as the similarity matching is valid at the time of retrieval, you can get the standard answer. Moreover, in this case there is no need to worry about the degradation of the original model's capabilities.

Answer 3

In Scenario 1, using high-quality documents for additional pretraining integrates the new information directly into the model, improving its general knowledge and generating more accurate responses. However, this process is computationally intensive and time-consuming. In Scenario 2, creating a vector database allows for efficient retrieval of relevant information by enhancing the prompt with similar text chunks, providing context-specific responses. This method is more flexible and easier to update with new documents. Generally, Scenario 2 might yield better results for specific queries as it dynamically leverages the most relevant information, whereas Scenario 1 improves overall model knowledge but might not be as targeted.