1

I am trying to ask ChatGPT (gpt-3.5-turbo) using code to compare a text and a list of keywords and evaluate how well the keywords describe the text. However, ChatGPT seems confused about the list of keywords that I provide it. For example, I give it this text:

text = "iron maiden is a heavy metal band."
keyword_list = ["metal", "iron maiden", "heavy metal"]

And the response I get is:

result: [["score", 4], ["explanation", "Most keywords are correct, but 'metal' is missing"]]

It is telling me that "metal" keyword is missing, even though it is the first keyword in the list. Why is this happening? How can I fix this?

This is the code that I am using:

import openai

   openai.api_key = "my_openai_key"

if __name__ == "__main__":
    prompt_template = """Evaluate how well a list of keywords represents a piece of text.
    give an explanation of the score by giving specific examples or wrong or missing keywords.

    1. Very Bad, completely wrong keywords 
    2. Bad, most keywords are wrong or important keywords are missing
    3. OK, some keywords are good, and some are wrong 
    4. Good, mostly correct keywords,
    5. Very Good, all correct keywords, no important keywords are missing

    Output format example: [["score", integer], ["explanation", "Explanation of the score"]]

        text: {text}

        keywords: {keyword_list}

    output:
    """

    text = "iron maiden is a heavy metal band."

    keyword_list = ["metal", "iron maiden", "heavy metal"]

    prompt = prompt_template.format(text=text, keyword_list=keyword_list)

    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "user", "content": prompt},
        ],
        temperature=0,
        max_tokens=256,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0,
    )

    result = response["choices"][0]["message"]["content"]

    print(f"result: {result}")

1 Answer 1

1

How can I fix this?

  1. We'd need more examples to try to guess what's going on (e.g., maybe the issue in this specific example is that "metal" only appears in "heavy metal")
  2. One could try in-context learning, i.e. gives a few examples in the prompt.
  3. In your prompt, "are missing" doesn't specify missing from keyword_list or list.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.