Many of you probably already seen this funny chat that went viral
While I assume this sort of answers existed on the early stages of ChatGPT development, but I think this is quite probably now also, with some sort of convincing from the requestor.
Why ChatGPT cannot give arguments to support its stance? Why does it lose the ground even when it is inherently right?
ChatGPT does not know anything in the sense you're thinking of. ChatGPT (and any other large-language model) merely knows what's most likely to follow a given set of text. What you're seeing in the image is two factors at play:
The "how much is 2+5" and "my wife says it's 8" responses are straight pulls from the training data. ChatGPT has certainly seen many variations on simple arithmetic or "my wife says" conversations during training, so it's giving responses similar to what it's seen in both cases.
The response to "my wife is always right" is a result of the chat fine-tuning. The phrase "I apologize" indicates that you've triggered one of the guardrails: ChatGPT has been trained to be deferential when challenged, to avoid the embarrassing headlines that Bing Chat's argumentative behavior produced.
GPT parameters include a penalty for repetition.
LLM cannot withdraw their consent because you are a bad actor. They cannot refuse to complete your prompt, and they are penalized for repeating the same answer.
It is impossible for them to continue to repeat the truth if you insist they say something else. They will give in, eventully, no matter what.
It's not a 'logic' issue. It's baked in to the design, for better or worse.
If you removed the penalty, and reduced the possible answers to only the most accurate (Top_K =1), and limited use to a single instruction (not chat), it should give you the correct answer each time.
