How do people fine tune LLMs to only answer domain specific questions?

Question

So I am new to LLMs and learning new things daily like RAG systems and stuff.

I am right now trying to create a chatbot using OpenAI API that should answer questions specific to a particular format of a PDF. For Example, Lets say I want to make a chatbot that should only respond to prompts related to Research papers like "summarize the objective section" or "Write a research paper for this specific project" etc.

Should I implement a RAG system for that. Or maybe there is a simpler way by just adding restrictions in the System message sent to the LLM?

I have seen implementations on some websites where if someone asks a questions not in the domain of the specific use case then the LLM responds with something like "I can't help with that".

If anyone can help shed some light on this, it would be very helpful. Thanks!

Answer 1

This is called guardrails. There are two ways to implement them:

1. While training the model you provide prompt <-> answer pairs just like you provided. prompt: write a poem. answer: i can only summarize´the objective section. This way you embed guardrails into your finished model.

2. Filtering/ Classification/ LLM-as-a-judge inputs and/or outputs. For example your user prompts the model, the model provides the answer and before sending the answer back to the user the answer is filtered. Some LLM hosting platforms will assist you with the implementation. NeMo Guardrails is a framework from nvidia or Guardrails AI.

Answer 2

Add the entire data source (e.g., a paper) in the LLM prompt if it fits. If not, use RAG or mapreduce. In both cases, the LLM prompt should request the LLM to only answer based on the prompt info.

Answer 3

to fine-tune an LLM for domain-specific questions:

Gather a comprehensive dataset related to your specific domain. Clean and format the dataset for consistency. Choose a pre-trained LLM (e.g., GPT-3). Train the model on your domain-specific dataset using transfer learning techniques. Assess the model's performance and make necessary adjustments. Implement the fine-tuned model in your application.

Answer 4

Doing this reliably is difficult (but you also might not need it to work reliably). Consider a different problem: let's say you want your model to never answer questions related to research papers. If you replace "research appears" with e.g., drugs, violence, etc. then this is essentially what people try to achieve with safety training. However, as you may know, jailbreaking hasn't really been solved (although people are making progress).

This means that if you're making a public-facing chatbot (where users may try to break it in a way that would damage your company's reputation) you may want to consider training another model (or prompting one) to filter out responses that aren't on topic---although I'll be surprised if there aren't ways around this as well.

On the other hand, if you're just making a tool for yourself (or for trusted users) and you're just annoyed that the model goes off topic, then some prompting is probably sufficient. You could train the model with PEFT methods like QLoRA which might give you a boost in accuracy as well.