RAG for sophisticated question-answering (Q&A) chatbots

Question

I am about to start my first ever hobby project with LLMs. I hope to develop a Q&A chat bot. I am following the documentation from here.

LLMs can reason about wide-ranging topics, but their knowledge is limited to the public data up to a specific point in time that they were trained on. If you want to build AI applications that can reason about private data or data introduced after a model’s cutoff date, you need to augment the knowledge of the model with the specific information it needs. The process of bringing the appropriate information and inserting it into the model prompt is known as Retrieval Augmented Generation (RAG).

Assume that, I have an LLM that was trained on salary slips of a certain organisation. Using RAG for salary slips of a different organisation for this LLM, I understand. But, suppose, I now want to use this LLM for tax returns. Are we saying that, RAG on tax returns with LLM based on payslips will "work"?

Can you help me understand - light on math and mild or heavy on code, please?

Answer 1

It depends on the LLM, RAG works even without the LLM having been trained in the data with the right prompt and a generalist LLM like GPT-4, GPT 3.5, Gemini, LLama 2, Mistral and etc. Being fine tuned in the data can improve RAG performance but it also works in most cases without this.

But the ability of an LLM to switch from different tasks and contexts depends on its general ability of doing this. And how good it will perform on it also depends on the LLM general performance summed with prompt enginnering, RAG implementation and etc.

For example, if you fine-tuned and GPT 3.5 on some data you have and want to use RAG to extend to other data is likely to work.

If it is a LLama 2 model, it depends on how the fine-tuning was implemented and if it was capable of retaining his previous capabilities from it.

So, the general answer is yes, the specific answer for you case depends in its specific details.