9

Here, "talk like a pirate" is just an example. For a more practical example, there have been a few questions here about asking ChatGPT to phrase an answer differently, for instance this one about getting it to stop apologizing. Often, this is done by appending "Please respond like a ..." to the query. For instance, the top answer in the linked post at time of writing is: "Respond as if you are an unapologetic assertive person for the rest of this conversation."

Various LLM benchmarks exist to measure how good LLMs are at answering questions correctly. I'm curious if performance on these benchmarks is affected (and if so, by how much) when this style of "please phrase your answer..." is used.

If I ask an LLM a question with a factual answer, will it get it wrong more frequently if I have told it to speak as a pirate or as an assertive person?

1 Answer 1

4

Various LLM benchmarks exist to measure how good LLMs are at answering questions correctly. I'm curious if performance on these benchmarks is affected (and if so, by how much) when this style of "please phrase your answer..." is used.

For the benchmarks that rely on automated evaluation metrics based on n-gram overlaps such as ROUGE, METEOR or BLEU, asking LLMs to output the answer in a style that is distinct from the style of the reference texts will decrease the performance numbers.

3
  • How do we know that? Are there some references?
    – A.L
    Jul 31, 2023 at 22:50
  • 1
    @A.L using a different style results in fewer n-gram overlaps. Jul 31, 2023 at 23:19
  • This is a correct answer haha, but doesn't really address the root question I was trying to ask. But good point :)
    – Kaia
    Aug 1, 2023 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.