As shown in the screenshot, the model outputs or leaks what appears to be some of the prompts used to train it. I couldn't reproduce the issue though. There were some other questions I had asked before asking "why are you named orca?".
As an individual, using OpenAI's API's are expensive, so I wanted to use models locally and access them via Python, using gpt4all's API. This error however has been a major disappointment.
This could also mean such errors can happen in commercial GPT's too. So we need to know what causes it and how it can be avoided.

Temperature: 0.7, top_k: 40, top p: 0.4, prompt batch size: 128, repeat penalty: 1.18, repeat penalty tokens: 64.

enter image description here

1 Answer 1


GPT-based chatbots work by predicting the text in a specially-formatted document that looks something like this:

Human: What is the capital of France?
Bot: As of 1789, the capital of France is Paris.
Human: Fascinating. Whose fault is that?
Bot: We could blame the Celts, or the Romans, or even geography, but King Clovis made Paris the capital when he took over Gaul, so I'd blame him.

This then gets passed through a computer program, which extracts the "Bot:" lines and stops the generation before it produces "Human:" lines – but this process relies on the model faithfully reproducing the separators. As a fundamental consequence of how GPT models work, sometimes they'll get the formatting slightly wrong, and then the program driving the model parses that malformed output incorrectly, and the illusion shatters.

Those probably aren't the prompts used to train it. Those are simply the model's predictions of what you will say next, because the model doesn't understand that it is not you, and the supervisor program that normally hides that fact isn't clever enough to spot the cascading failure in progress.

There is no way to avoid this: it is simply how the technology works. Predictive text models are not the universal hammers they're marketed as: they are the wrong tool for the job, most of the time they're used. (In ten years' time, hopefully, that will have changed, because people won't be using them as much.)

  • Thanks. I did always feel that neural networks were an incorrect tool to model intelligence. The C.Elegans does not have activation potentials and the sea sponge does not even have neurons. Intelligence is something entirely different. In this specific case, I guess we can have another GPT bot which checks the outputs to see if it is problematic.
    – Nav
    Dec 13, 2023 at 18:20
  • 1
    @Nav You'd want a classifier, not a transformer. Even then, that's not a solution, just a patch job: there would be some other failure mode that you'd have to bolt on a solution for, and then another… Best to only use it where it's allowed to be fallible, and where you can justify the resource expenditure.
    – wizzwizz4
    Dec 13, 2023 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.