4

Here is my script:

from transformers import AutoTokenizer, AutoModelForCausalLM   

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf")

prompt = """
CONTEXT: Harvard University is a private Ivy League research university in Cambridge, Massachusetts. 
Founded in 1636 as Harvard College and named for its first benefactor, the Puritan clergyman John Harvard, it is the oldest institution of higher learning in the United States. Its influence, wealth, and rankings have made it one of the most prestigious universities in the world.

QUESTION: Which year was Harvard University found?
"""

input_ids = tokenizer(prompt, return_tensors="pt").input_ids
outputs = model.generate(input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))

Here is the output:

<s> 
CONTEXT: Harvard University is a private Ivy League research university in Cambridge, Massachusetts. 
Founded in 1636 as Harvard College and named for its first benefactor, the Puritan clergyman John Harvard, 
it is the oldest institution of higher learning in the United States. Its influence, wealth, 
and rankings have made it one of the most prestigious universities in the world.

QUESTION: Which year was Harvard University found?
ANSWER: Harvard University was founded in 1636.</s>

You can see Llama-2 includes the input prompt for the output. Is there any way to remove the input prompt from the output?

2

3 Answers 3

1

what i do is if it contains "Answer:" it only prints what follows. but if you find a better solution please let me know:

if "Answer:" in output_text:
                start_index = output_text.index("Answer:")
                print(output_text[start_index:])
            else:
                print(output_text)
0

You can use the generate() method and pass the input_ids directly without including the prompt text. This will generate the continuation without repeating the prompt.

For example:

input_ids = tokenizer(prompt, return_tensors="pt").input_ids 

outputs = model.generate(input_ids, max_new_tokens=50)

print(tokenizer.decode(outputs[0]))

This will generate just the continuation text without the original prompt. The key is to not include the prompt text when encoding the input. Pass just the input_ids to generate().

-1

you're making it print its thought process, essentially. it should only print the answer. changing the last bit to the below should work:

answer = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(answer)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.