9

While I understand that GPT-4 does not use logic in the same way that a human does, is there some way to have ChatGPT "explain" itself?

Improve this question
2
  • 3
    This is one of the complaints about AI -- it can't explain how it came up with an answer, and neither can we.
    – John Gordon
    Jul 30 at 4:01
  • 3
    This is some kind of a meta-technical question. The LLM can explain, how it functioning, how it is programmed and how technically it works. But it can not "explain" in the sense of, "Where does my thoughts originate and how and why I (LLM) came to this conclusion?" We, as humans, have the same difficulty to answer this question on a meta-level. What science showed in the meantime was, which neurons in the hidden layers are showing strong activation in regards to the context of the prompt. The same, we have regions or bounds for sensorics or non-sensorics activities in our brain.
    – PriNova
    Jul 30 at 17:36

4 Answers 4

Reset to default
24

Yes and no.

You can certainly prompt ChatGPT (or any other LLM) to produce an explanation for why it's generating an output. In some sense, this could be seen as "the" explanation for why it generated the output, since the request will tend to change the output so that it matches the explanation.

In another sense, the explanation is complete nonsense: it bears no relationship to the actual mechanism of generating an output. Rather, it will be a reflection of "think step by step" or "explain your reasoning" examples in the training data. If you want to know how a LLM generates an output, you need to analyze the underlying neural network as it's running, something we don't know how to do for any but the most trivial networks.

Improve this answer
1
  • 2
    It is worth pointing out the parallel to a human response to this question. If you ask a devout christian, they may say "it came from the will of God". If you ask a neuroscientist, they may say "it was the result of electrical pathways in my brain". If you ask a philosopher, they may say "it was derived from this logical chain of thought". They may all be very intelligent people, give a very detailed explanations of the process, and be very confident of the answer. Nonetheless, that won't make them trustworthy explainers of the process which produced their answer.
    – Luiz Martins
    Jul 30 at 14:39
10

When presented with a prompt or question, the LLM analyzes the input and makes probabilistic predictions about which words are most likely to follow based on patterns and correlations learned during training. The training data includes millions of passages of text from books, websites, and conversations that are used to build statistical models of language.

So for this question about explaining answers, the LLM can not internal reason step-by-step about how to answer. Rather, it generated an answer by sequentially predicting the most likely next word/token that would result in a coherent explanation, based on the language patterns it had learned.

Thus, in general, no LLM can reason or explain its own behavior of analyzing the responses it makes because it has no internal symbolic logic. They can be asked to apply self-criticism to a given response. And by critiquing the explanation, they can find holes in transparency, such as identifying places where the probabilistic predictions are more hidden. The training has allowed them to produce reasonable explanations, but they don't have a real understanding of the internal mechanisms.

Improve this answer
8

GPT can produce a simulacrum of an explanation, but this is not necessarily a truthful explanation of how it actually reached its answer.

By way of illustration, here's GPT-3 responding to a long multiplication problem. Previously in this session I had asked it to "show working", so it gives me a worked answer here.

Screenshot of a GPT-3 session. The user asks "what is 123*321?" GPT-responds with a worked long multiplication. The first of the partial products is shown as "963 (123 × 1)" and the screenshot has been edited to mark this error with a yellow rectangle. The working then concludes with the correct answer, 39483.

39483 is the correct answer to the multiplication, but there's an obvious error in the working. 123 times 1 is not 963! Adding 963 to 2460 to 36900 would give 40323. But instead it shows the sum of these as 39483, the correct answer to the original question.

By trying the same approach with different 3-digit numbers you can get many examples of the same behaviour: the final answer is often (not always) correct even with incorrect working along the way.

Unless one supposes that by great good fortune its errors keep cancelling out to yield a correct answer, this should make it pretty clear that GPT isn't actually getting its answer by following the working it presents. If anything, it appears to be getting the final answer first (I suspect by learning patterns along the lines of "12* times 32* often gives an answer beginning in 39, *23 times *21 often gives an answer ending in 483") and then interpolating the requested working, rather unreliably.

This example was generated via GPT-3 rather than GPT-4. It may be a little harder to catch GPT-4 out with this particular approach, but I'm not aware of any reason to believe the changes between GPT-3 and GPT-4 would have made its explanations more truthful.

Improve this answer
3
  • The fact that it even CAN do multiplication is basically magic. Given that it is a model for language...
    – kutschkem
    Jul 31 at 11:28
  • 1
    @kutschkem GPT-3 is somewhat unreliable for multiplying 3-digit numbers and very unreliable for longer than that. I picked a case where it found the right answer for purposes of this post, but it often fails. It pretty much always gets the last two digits exactly right, and the value right within about 1%, both of which can be achieved by memorizing a 100x100 lookup table of results it will undoubtedly have seen in training. But for longer products the middle digits are extremely unreliable, and sometimes even the first digit wrong.
    – GB supports the mod strike
    Aug 1 at 6:05
  • I see, at least that seems less like magic and more in line with what I would expect a language model can do.
    – kutschkem
    Aug 1 at 6:12
7

Yes: ask it to "think step by step and explain its reasoning".

This is a common technique to make it produce more sensible answers, rather than jump to wrong conclusions. It's called Chain of Thought (CoT) prompting. You can read articles about it eg at arXiv: https://www.google.com/search?q=arxiv+cot

Improve this answer
13
  • 7
    Does it actually follow the steps it gives, or is it explaining its actions after the fact?
    – dave20
    Jul 28 at 16:10
  • 7
    @dave20 your instincts are correct -- it's the second one. It can't do the first one.
    – Glenn Willen
    Jul 28 at 19:07
  • 4
    @GlennWillen Not so fast. If the chain of thought comes before the answer, it can follow the steps. CoT prompting improves performance significantly over naive prompting (arxiv.org/abs/2201.11903).
    – Omegastick
    Jul 28 at 23:38
  • 8
    That means it's making the answer consistent with the steps, but that doesn't mean it followed the steps.
    – Charles Duffy
    Jul 30 at 1:05
  • 2
    @CharlesDuffy At least with GPT3, this issue can often be demonstrated by asking for a worked solution to something like "what is 123 times 321?" Sometimes GPT will produce the right answer but with errors in the working that should never have led to the solution it gives. Clearly it's starting with the answer (which would essentially be memorised for numbers that small) and then fabricating a chain of reasoning which has nothing to do with how it actually got there.
    – GB supports the mod strike
    Jul 30 at 5:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.