4

Background: I am currently working on an NLP project using GPT 3.5 via the OpenAI python API. Engineering good prompts seems to be the most critical part of my pipeline so far. Crucially I have long input texts, for which I want to get numerical sentiment scores for a variety of different questions in a ZERO SHOT setting. There are many possible applications such as predicting nuanced scores on different aspects (plot, acting, staging etc) of a theatre show from a long detailed review; or the sentiment of a twitter response to a new political policy (emotional objections, economic objections, moral objections).

Question: What is best practice to extract reliable numerical sentiment scores from GPT models?

What I've tried: Looked for guidance on the OpenAI cookbook and explored many associated links. It seems like existing suggestions for sentiment scoring usually use embeddings in a setting where a small number of labelled samples are available. Following heuristics from various prompt engineering sources I have used a pipeline of iteratively asking for detailed summaries and then more concise summaries culminating in a single sentence and finally a score. This has yielded impressive results; indeed I suspect such a 'prompt-pipeline' may yield more sophisticated sentiment scores than fine-tuning approaches, unless one has the luxury of extremely large amount of labelled data. However, converting these final sentences to numeric scores is still somewhat unreliable. I couldn't find specific advice online. What is best practice?

1

1 Answer 1

2

To address converting the final sentences to numeric scores being unreliable, you could use logits_bias to restrict the model to only predicting certain tokens.

You can use OpenAI's tiktoken library to find token IDs:

enc = tiktoken.encoding_for_model("gpt-3.5-turbo")
print([enc.encode(x) for x in "1234567890"])
# [[16], [17], [18], [19], [20], [21], [22], [23], [24], [15]]

Integers up to "999" are each represented by a single token, higher numbers and decimals get split up into multiple tokens.

Then, using the OpenAI Python API, adding a bias of 100 will ensure exclusive selection of one of those tokens:

response = openai.ChatCompletion.create(
    engine="gpt-35-turbo",
    messages=[{"role": "user", "content": "How many fingers are on a hand?"}],
    temperature=0,
    max_tokens=1,
    frequency_penalty=0,
    presence_penalty=0,
    stop=None,
    logit_bias={16: 100, 17: 100, 18: 100, 19: 100, 20: 100, 21: 100, 22: 100, 23: 100, 24: 100, 15: 100}
)
print(response["choices"][0]["message"]["content"])
# 5

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.