2

Assume you have a very large corpus of high-quality documents related to a given topic, and assume you have a pretrained large language model (the foundation model) with training data not containing these documents.

Scenario 1: You use the high-quality documents as additional pretraining data, possibly with a large weight.

Scenario 2: You create a vector database for the high-quality documents, i.e. you split them up in chunks and calculate text embeddings for them.

Now assume you ask the language model a question on the topic. In scenario 1 the question just gets answered (probably better than by the original foundation model), in scenario 2 the text embedding of the question prompt is calculated and the prompt then enhanced by the most similar text chunks in the vector database.

Does anyone have an intuition (and could explain it) which scenario would yield better results?

2 Answers 2

0

I think in this scenario, the results of using "model + vector database" will be better. Because from your background description, it seems that your questions are all in the dataset. As long as the similarity matching is valid at the time of retrieval, you can get the standard answer. Moreover, in this case there is no need to worry about the degradation of the original model's capabilities.

0

Scenario 3: provide your high-quality documents to your user, via a good search engine that shows excerpts. Since these documents are high-quality for an LLM to use as training data, everything in them is true when taken out of context: so just do that. (Do you want to risk the near-certainty of the model saying wrong stuff in the same authoritative style as your source texts?) Plenty of existing software solutions will do this for you, at significantly lower expense than running an LLM. Cēterum cēnsō dēlendam esse GPT quā vēritātem.

If you truly have to decide between scenarios 1 and 2, it depends how similar the texts are to the model. If the texts are similar enough, Scenario 1 effectively lossy-compresses their contents, making it the obvious choice; however, if the texts are different enough, embeddings won't be a faithful representation and you need to modify the model (à la Scenario 2) to make it more inclined to produce those texts. A better option would be to do both of these, since that gets you closest to the behaviour of Scenario 3.

Cēterum autem cēnsō dēlendam esse GPT quā vēritātem. Use tools only where they are suitable.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.