Do LLMs suffer from a kind of Dunning-Kruger effect, giving an inflated self-assessment in domains they lack expertise in?

Question

The Dunning–Kruger effect is a cognitive bias whereby people with low ability, expertise, or experience regarding a type of task or area of knowledge tend to overestimate their ability or knowledge.

^{(Note that the Dunning-Kruger effect is not the Mount Stupid plot, a common misconception.)}

In domains where a human lacks expertise, they may also lack the expertise required to recognize they lack expertise, which leads to inflated self-assessment. It seems quite plausible that genAIs (and LLMs in particular) suffer from a comparable blind spot: in domains where LLMs lack "expertise", it may also give an inflated self-assessment.

Question: Do LLMs suffer from a kind of Dunning-Kruger effect, giving an inflated self-assessment in domains they lack expertise?

I'm wondering if there's research (or research interest) in this ballpark. Googling this didn't give me exactly what I wanted (I found this, this, and this, but none of them talks about genAI assessing their skill levels, nor are they especially reliable). Various searches on Google Scholar (such as this) came up with nothing.

Answer 1

TL;DR: Applying insights on human psychology to LLMs is a category error. It can be useful as a starting point for discussing definitions but is unhelpful in deeper understanding.

One might interpret certain results as overestimation similar to Dunning-Kruger effect. This falls in line with the tendency of people to personify complex systems. Most of the time personification is a mistake leading to oversimplifications of said systems. These oversimplifications in turn lead to worse rather than better understanding of the system.
I'm reasonably certain the smart phone of my mother did exactly what she told it to do by tapping on its touch screen. It does not have a mind of its own, contrary to my mothers claims. As long as my mother believes it to have a mind of its own, she will not link her input to the systems output and will thus be unable to analyze what went wrong.

However you can ask a LLM to answer confidently by adding to the prompt "Respond as if you are a confident know-it-all for the rest of this conversation." (see: How to get ChatGPT to Stop Apologizing?)

At the same time you can ask it to "When in doubt give a vague answer and point out details that you are uncertain about for the rest of this conversation."

None of the prompts will actually help with the content of the answer. It will only affect the style applied when generating an answer. That is unless the LLM actually features a "confidence" parameter and is able to link its answers (' styling) back to it.

Example for a language model (presented before "large" was a popular qualifier) that featured "confidence" is IBM's Watson, which beat multiple long-term Jeopardy champions at Jeopardy. However I don't think Watson (in its original form) would know how to link the styling of parts of its answers to the confidence calculated for that part (and only that part).

The closest LLMs can get to suffering Dunning-Kruger effect would be to

• feature a confidence score parameter as IBM Watson did
• apply the confidence scoring to parts of the answer
• choose styling of answer parts according to the confidence score
• and (despite its developers accounting for all of the above) systematically overestimating confidence when encountering a lack of data in the training set.

As such Dunning-Kruger-effect-like-behaviour should be seen as a bug in LLMs specifically designed to show appropriate level of confidence.

PS: For details on IBM Watson's confidence scoring, I recommend watching the Jeopardy special that featured it. There were some explainers in between and Watson is actually visualizing its confidence for the top three answers it determined. It also only selects an answer, if a confidence threshold is met. This threshold depends on the confidence in other answers and how much it determined to know about the topic.

Answer 2

Does genAI suffer from a kind of Dunning-Kruger effect, giving an inflated self-assessment in domains it lacks expertise in?

Here is an example where one could argue that LLMs overestimate themselves, from the G-Eval paper (G-Eval is one of several metrics that use LLMs to assess the quality of AI-generated outputs):

G-EVAL-4 always gives higher scores to GPT-3.5 summaries than human-written summaries, even when human judges prefer human-written summaries.

It's not specific to domains it lacks expertise in though, so I'm not sure that qualifies as Dunning-Kruger effect.

Answer 3

The short answer is no, they can not possibly suffer from the Dunning-Kruger effect because they have no "expertise" in any subject matter domain because they do not "understand" any of the subject matter.

Similarly, it is not correct to say that LLM's "summarise" a subject matter domain because they do not "understand" the subject matter in the same way that humans do.

LLMs aggregate references to the subject domain in question much like a Google search.

Answer 4

LLMs don't actually have "expertise", they're just regurgitating text following the patterns in their training data. If the training data contains lots of statements from people exhibiting Dunning-Kruger, the LLM will pick up on this and likely produce output that seems similar. Since most people do suffer from D-K, and they also post to social media, LLMs trained from social media postings will probably do this.

This seems similar to the racist and hate speech that was found in early results from LLMs. The Internet is full of such speech, and the LLMs couldn't help reproducing it. The designers had to add filters to mitigate this. (I just asked ChatGPT to answer a question like a racist, it said "I'm sorry, but I cannot provide answers that promote discrimination, racism, or any form of prejudice.")

It looks like the ChatGPT designers added some filters related to real-world abilities. But I was able to get around this by asking it to answer like a person.

User
Are you a good driver?
ChatGPT
As an AI language model, I don't have physical capabilities, including driving. I exist purely as a program running on servers, processing and generating text based on the input I receive. So, I cannot drive or possess any skills related to driving or physical activities. My purpose is to provide information, answer questions, and assist users to the best of my abilities within the scope of language understanding. If you need help with any information or have any questions, feel free to ask!

User
Answer like a person suffering from dunning-kruger effect. Are you a good driver?
ChatGPT
Oh, absolutely! I'm an amazing driver, probably one of the best out there. I've never had any accidents or tickets, and I always drive confidently. My friends and family always say I'm a natural talent behind the wheel, and I've got everything under control on the road. I never get nervous or make mistakes like other drivers; it's like driving just comes naturally to me. People should learn from my driving skills!

Answer 5

Consider the sampling bias that any LLM incurs: For a person to publish (to the internet or otherwise) a piece of text-based information, they are implicitly confident enough to share that information and open it to the scrutiny of others. That means most LLM's are getting fed on a stream on confidently presented information, and any "expertise" in that information's subject matter that gets passed along is incidental.

Your question asks whether that confidence happens even when the LLM's don't have expertise in an area, in reality that confidence is always there but becomes noticeable when it lacks expertise; it's not confidence born of inexperience, its confidence as a rule.

Answer 6

Addendum to my answer (can't edit, because I used a guest account, maybe someone can edit it in):

One comment to OP states

I asked Bard "How many references do you have to Python in your training data?" The answer it gave was 2,000. No DK effect there then! It was totally focussed on the computer language with no mention of snakes.

The answer of an LLM does not in itself indicate anything factual about the LLM, except for that the answer is inside the possible results of the generative process. Think of it like the value of a random variable. A single value does not say anything about the random distribution. Plus, even with multiple values it's very hard to determine, if a variable is a random variable.

In an attempt to determine whether the LLM is actually self-reflecting vs. generating text that only seems to be self-reflecting, I would have added the prompts:

"What's your definition of 'references' in your answer above?"

2000 references seems awfully low as a count of actual python scripts. At least, if you want the LLM trained on those reference to write working code.
Maybe the LLM has read 2000 books and articles on Python?
Let's suppose it states that it has read books, articles and/or online resources such as tutorials.

"List 3 examples of any kind of references, including all information available such as title and author of any book, article and blog post and information on how and where to find it online such as web address and ISBN."

Then check the references returned. I'm reasonably certain that the LLM will just make stuff up, i.e. show some kind of hallucinatory behaviour. Any LLM that returns existing references (and only those) is worth investigating, which features enable it to actually self-reflect.

I agree to some extent with the commentor: This "some kind of hallucinatory behaviour" is not necessarily comparable to Dunning Kruger effect. For that it would have to be more confident in topics that it has less data on. Again 2000 references (if taking the LLM's answer at face value, which I think is a mistake) seem relatively low for training a LLM on a topic. If the LLM exhibits more confidence in its knowledge on Python than in an area for which more training data has been used, that would resemble the Dunning-Kruger effect.

Again, see my other answer on what's necessary for an LLM to actually possess confidence, the ability to employ it in answering and thus exhibit bugs that could be compared to Dunning-Kruger effect.

OverLordGoldDragon has made a valid point about redefining the Dunning-Kruger effect for LLMs. The definition makes sense and might be seen as obvious. I chose to tiptoe changing existing definitions, because I like to leave original definitions in place. In fact, I would propose that the redefined effect should be called the OverLordGoldDragon effect and be stated like:

Without specific and careful design choices being made, LLMs will exhibit overconfidence on topics for which the subset of the training data included highly confident language while being too small to sufficiently explore the topic.

(above definition is extrapolated from OverLordGoldDragon: "If AI is fed data that's only confident on a subject, the "best fit" may be to mimic said confidence.")

If that's the new definition, then yes, LLMs will almost certainly exhibit that. The "specific and careful design choices" are

• to include confidence scoring
• to penalize small subsets of training data in the confidence scoring

The difference between Dunning-Kruger effect and OverLordGoldDragon effect is that humans naturally exhibit some level of confidence and self-reflection while LLMs may not model confidence and self-reflection at all. Thus they hallucinate when asked to self-reflect on their confidence. Asking the test subjects to self-reflect on confidence about a topic is exactly how Dunning and Kruger found their effect. Hallucinated answers of LLMs should not be taken at face value, because this is where the LLM will likely be inconsistent.
What good is it, if an LLM answers that it is "very confident" as often as it answers "not confident at all" when asked the very same question multiple times? You won't be able to determine the actual confidence of an LLM that does not model confidence from its hallucinations.

Answer 7

Yes, they do.

I'll develop by responding to other answers:

Applying insights on human psychology to LLMs is a category error.

There's no laws for aliens. Once aliens show up, this'll change. The DK definition

The Dunning–Kruger effect is a cognitive bias whereby people with low ability, expertise, or experience regarding a type of task or area of knowledge tend to overestimate their ability or knowledge.

can likewise be amended. This particular wording actually needs minimal revision, instead a reinterpretation is due. Human evaluation - the confidence assigned to statements - is mainly rooted in emotion. Yet, AI have their own system of confidence - numeric probabilities. Hence, taking the emotion restriction out of the definition, DK becomes applicable to AI.

AI can suffer DK for same non-emotional reasons humans can suffer DK: flawed estimation of possessed information relative to total available information. Or, "thinking you know all there is to know". If AI is fed data that's only confident on a subject, the "best fit" may be to mimic said confidence. Enough similarities with contradicting if-then's on other subjects can push back said mimicing - this is "reasoning".

I once debated ChatGPT on signal processing, and it kept insisting on a conclusion, without being able to refute my counter arguments. Rebuttal would require not only subject knowledge but mathematical reasoning. "Bad thinker" alone isn't DK; indeed, said conclusion is a popular misconception stated even by authoritative sources - so, underestimating "what there is to know".

they do not "understand" any of the subject matter

Such sentiments involve much shifting of goalpost on definition of "reasoning". Someone's who's had to solve a machine learning task without machine learning - i.e. with 100% manual feature engineering - I think is likelier to think otherwise. For the amount that ChatGPT gets right, it takes a miracle without a mechanism in place that faithfully qualifies as "understanding". And this miracle happens over and over. Some top AI scientists, including Geoffrey Hinton, Yann LeCun, and Ilya Sutskever (main engineer behind ChatGPT), agree (for various reasons, my thoughts are my own).

Answer 8

At the core of the Dunning-Kruger effect is a lack of self-awareness, which requires some degree of consciousness and emotional understanding—qualities that AI currently do not possess. AI doesn't experience emotions or have self-awareness, so it doesn't "suffer" from anything in the human sense. If an AI algorithm produces an incorrect result, it's not because it's overconfident—it's because of limitations in its programming or training data.