I've seen this, this, this, and this. But my question is different.
Whether I use ChatGPT's API or any of the LLM's supported by GPT4All, when I deploy my GenAI app for people to use, I want it to answer only questions based on the dataset I provide it.

Eg: If I build an app for lawyers, I want them to be able to ask my GenAI, questions about a specific case or legal provisions, from a database of knowledge that I provide on my server. If the User starts asking my app questions about Marilyn Monroe's husband or anything else that's not in the database that I provided, the app should respond with "Sorry, I've been instructed not to answer questions outside the legal database".

enter image description here

SkyPilot's AI was trained to respond with "I'm sorry, but as an AI assistant for SkyPilot, I'm tuned to only answer questions about SkyPilot. I don't have the ability to provide information about why the sky is blue. If you have any questions about how to use SkyPilot or its features, I'd be happy to help with that!" on being asked "Why is the sky blue". But even then, it was possible to make it do tasks like writing a story or a poem, while pretending that it was about SkyPilot.

enter image description here

Obviously I can't train my own LLM based on only the data in the database, because then the LLM won't have the kind of language skills that current LLM's have. So is it possible to use LangChain or any other tool to create a foolproof method of answering questions based on only a select database? Or is that silly to expect, because of the way an LLM functions?

  • Please take a look to genai.meta.stackexchange.com/a/119/12
    – Wicket
    Jul 29, 2023 at 17:48
  • Answers to Are there powerful text generators that preserve attribution? might be answers to this question, and vice versa. I think they're distinct questions, though.
    – wizzwizz4
    Jul 29, 2023 at 22:05
  • The other problem is that even if they ask questions that are within its scope, it might answer them incorrectly based on its understanding of its training data instead of strictly on the dataset you've provided.
    – endolith
    Aug 2, 2023 at 15:19
  • 1
    @endolith: ok, that's a deal-breaker. I guess then the approach would be to use LangChain to use something like MongoDB to cross-verify if what GPT generated matches the document.
    – Nav
    Aug 3, 2023 at 10:52
  • 1
    @Nav I think providing citations with direct quotes would help a lot. github.com/blockpipe/BlockAGI attempts to do that, but the citations are sometimes incorrect. Or clever prompt design with multiple AI agents checking the veracity of each other's output or something like that.
    – endolith
    Aug 3, 2023 at 19:45

3 Answers 3


There are a few approaches you could take to constrain an AI to only answer questions based on a specific dataset:

  • Fine-tuning: Take an existing large language model like GPT-3 and fine-tune it on your legal dataset. This adapts the model to your domain while retaining its general conversational abilities. You would likely still need to implement some conversational scoping as well.
  • Retrieval-based: Build an information retrieval system that only indexes and searches your legal dataset to find answers. The chatbot would query this search engine to get responses. The chatbot itself could still be a general conversational model. Called Multi-Agent-System.
  • Modular/pipeline: Use a general conversational model for natural language understanding and a separate module that queries your legal knowledge base to generate responses. This keeps the components separate. Called Mixture-of-Experts (MOE)
  • Conversational scoping: Regardless of the main chatbot implementation, you will likely want to implement conversational scoping techniques like: allowing only certain topics, redirecting off-topic questions back to the domain, saying "I don't know" for out-of-domain questions, etc.

I expand more technically on "Conversational scoping":

You can directly manipulating the logits output by a language model. This is another technique that can help scope the conversational responses.

The logits are the raw scores that a language model produces for each token before picking the token with the highest score. By modifying these logit values for certain undesired tokens before selecting the response, we can steer the model away from tokens we don't want.

Some ways to accomplish this:

  • Zeroing out logits - Set logits for certain tokens to a very low value like -1e9 so they will never be selected. Can do this for topics, keywords, etc.
  • Weighted masking - Similar to above but rather than zero out, apply a negative weight to reduce some logits. Allows some probability.
  • Increasing desired logits - Boost logits for approved tokens to increase their chances of being generated.
  • Adding penalties - Subtract a penalty value from logits matching unwanted tokens.
  • Forcing replies - Set logit for special token like [REDIRECT] very high to force it to be used.
  • Blocking n-grams - Zero out sequences of prohibited n-grams.

The main caution is that directly manipulating logits can sometimes lead to unnatural or non-fluent responses if done too aggressively. So it takes some trial-and-error to tune the modifications. But used judiciously, it's a powerful way to shape the allowed responses.

  • These should each be separate answers, not one answer with multiple potential solutions.
    – endolith
    Aug 2, 2023 at 15:20
  • @endolith For some problems exists more than one solution. This depends strongly on the use case or issue to be solved. I gave the user several solutions to choose from. This benefits to his/her knowledge and enhancing the solution space. This opens up perspectives maybe overseen and may benefit for other future use-cases, too.
    – PriNova
    Aug 3, 2023 at 11:50
  • Yes, and those solutions should be posted as separate answers, so they can each be commented on or voted for individually. meta.stackexchange.com/a/25210/130885
    – endolith
    Aug 3, 2023 at 19:46

You can't restrict an LLM's answers either through prompting or through training. Restrictions in the prompt can be overridden by the asker being sufficiently clever in wording the question (for example, by asking in Japanese when the restrictions are written in English). Limiting the training data won't work because an LLM only knows how to predict the continuation of its input, not any domain knowledge. An LLM with a restricted training set will still attempt to answer questions outside its domain, it just won't do a very good job of it.

You have two options for constraining the sort of questions that can be answered:

  1. Filter the input. This could be anything from a simple keyword filter to a machine-learning model trained to classify questions as "good" or "bad". The idea is that out-of-domain questions never reach the LLM.

  2. Filter the output. Inspect the answer produced by the LLM, and reject it if it appears to be out-of-domain. For example, if you want the LLM to answer questions about legal precedents, you can reject any answer that doesn't contain a valid case citation. This has an advantage over the input filter in that you don't need to worry about the LLM producing an out-of-domain answer to an in-domain question.


Prashanth, CEO's Stack Overflow Inc., this week announced OverflowAI with promises to deliver content based on Stack Overflow. Considering this and how much you might trust Stack Overflow as a software developer using AI technologies, it might not look silly to expect that you could constrain your app to use a specific dataset.

I'm not saying that this can be done with OpenAI ChatGPT API and GPT4ALL; you might have to use another GPT foundation model than those available on these tools, maybe you will have to combine it with other technologies, or even you might have to implement your LLM built on your target dataset and complementary content that could help on writing coherently and grammatically correct.

I suggest you post a follow-up question about the state of the art of LLM and their capabilities in Artificial Intelligence Stack Exchange. Also, I suggest you learn about related offerings from major vendors like Microsoft and Google. Below there are some starting points:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.