Questions tagged [llama-2]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
37 views

Ollama GPU Support

I've just installed Ollama in my system and chatted with it a little. Unfortunately, the response time is very slow even for lightweight models like tinyllama. It seems that Ollama is in CPU-only mode ...
Jeron Baffom's user avatar
0 votes
0 answers
21 views

Cannot find a prompt to generate a quiz in CSV format

I want to generate quizzes and obtain the output in CSV format using llama2. I want to obtain something like this: 1;What is the primary function of the mitochondria in a cell?;a) To synthesize ...
Duck's user avatar
  • 101
1 vote
0 answers
22 views

What could I do to further reduce my finetuning time of 67 hours with the following configuration?

I am taking a large text dataset and fitting them into 19k rows. Each row is exactly 4096 tokens (not characters), which is the maximum context length of llama 2 7B. GPU: 80GB My other configurations ...
Maximos's user avatar
  • 43
3 votes
1 answer
50 views

What is the efficient way to tokenize a long string?

I have a really long string. How can I efficiently identify the boundaries of a fixed token length in the text? For example: text = "Quick silver brown fox jumped over the hedge" ...
Maximos's user avatar
  • 43
3 votes
2 answers
126 views

Why the Llama 2 language model keeps printing random garbage text when certain prompts are used?

On the website https://llama2.ai/ you can test the llama 2 model. But when I ask certain questions (or with some prompts) it sometimes prints random nonsense text. I tried reducing the "...
Poseidon of Milos's user avatar
1 vote
0 answers
278 views

GPU vRAM minimum requirements for multiple GPUs running a 70B Llama 2 model at 4 bits

Backround I would like to run a 70B LLama 2 instance locally (not train, just run). Naively this requires 140GB VRam. Quantized to 4 bits this is roughly 35GB (on HF it's actually as low as 32GB). I'd ...
RabidMutant's user avatar
1 vote
1 answer
77 views

Why LLMs are poor when recognizing weekdays for specific dates?

I asked the LLaMA V2 LLM using this site: https://llama2.ai/ - for the respective weekdays for February 29 for leap years since 2000. Here is the response: I'm happy to help! Here are the respective ...
Poseidon of Milos's user avatar
-1 votes
2 answers
277 views

What GPU to run llama-2 13B locally?

I've installed llama-2 13B on my machine. While it performs ok with simple questions, like 'tell me a joke', when I tried to give it a real task with some knowledge base, it takes about 10-15 minutes ...
Ya.'s user avatar
  • 101
3 votes
1 answer
221 views

Llama2 Vocab contents

since, Llama2 is multi lingual model and it supports multiple languages, including English, Spanish, French, German, Italian, Portuguese, and Dutch. Vocab Size of Llama is 32K. How to know out of 32k ...
Vinay Sharma's user avatar
3 votes
0 answers
195 views

Poor concurrency and perf on multiple GPUs VM (running CodeLlama locally using Ollama)

I am trying to figure out if self hosting CodeLlama on a sufficiently powerful multi-GPU machine can be cost effective for my specific product needs. However, when I go from 1 GPU to 4 GPU VM, I see ...
Shripad K's user avatar
-3 votes
1 answer
1k views

Best realistic story telling LLM?

I've been using WizardLM-SuperCOT30bUncensored for quite some time now, and I'm wondering if there is anything better these days for stories? Because apparently this model is cool, but I can see that ...
Marcin Jasny's user avatar
4 votes
1 answer
122 views

Is the generated code by Code-Llama and LLama-2 models licensed somehow or it has copyright issues?

I've been struggling with finding info about this on the official docs. I would like to use Code-Llama for pair programming but I don't know if this AI generated code is licensed somehow or I could ...
Omar Garduza's user avatar
3 votes
1 answer
152 views

Do Llama-2 and Code-Llama models collects my code?

As you know, Open Ai states that data collected through chatgpt is used to improve their models, but by using the OpenAi’s api, this is not stored or collected: https://openai.com/security I would ...
Omar Garduza's user avatar
1 vote
1 answer
48 views

how to do a quick pilot run when pre-training a large language model?

As I know, it will take several months to pre-train an LLM like chatGPT. I am wondering, what is the industry standard practice to do a pilot run to make sure that everything is correct (as it will be ...
Dmitry J's user avatar
1 vote
0 answers
156 views

Inference INT4 ONNX version of LLAMA-2 very slow on google colab

I am using the INT4 quantized version of Llama-2 13B to run inference on the T4 GPU in Google Colab. from optimum.onnxruntime import ORTModelForCausalLM from transformers import AutoTokenizer, ...
k-c's user avatar
  • 11
1 vote
0 answers
33 views

Get Llama-2 Rotary Embeddings

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the rotary embeddings and how can I ...
Christian01's user avatar
0 votes
1 answer
823 views

How would I add a knowledge base to a llama prompt?

I know that the prompting format for LLAMA 2 looks like this: <s>[INST] <<SYS>> {your_system_message} <</SYS>> {user_message_1} [/INST] {model_reply_1}</s><s>...
user2741831's user avatar
2 votes
1 answer
1k views

How to prompt Llama2 for text classification?

Here is my script: from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf") model = AutoModelForCausalLM....
etang's user avatar
  • 193
4 votes
3 answers
2k views

How to just get the answer from Llama-2 instead of repeating the whole prompt?

Here is my script: from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf") model = AutoModelForCausalLM....
etang's user avatar
  • 193
0 votes
0 answers
37 views

Reversible Transformer and LSH vs. SOA Models

In the DeepLearning.ai specialization on Natural Language Processing, the instructor claims that the Reformer, using LSH and reversible layers, is able to ingest context lengths of 1M+ tokens and ...
mrplants's user avatar
  • 101
1 vote
1 answer
1k views

How to prompt LLMs to get a concise answer for a question?

I use LLMs for QA tasks. The following prompt sent to Llama-2-13b-chat-hf: Give a precise answer to the question based on the context. Don't be verbose. CONTEXT: ..... QUESTION: what is the commission ...
etang's user avatar
  • 193
2 votes
2 answers
124 views

Embeddings: How can I find out, that I'm out-of-vocabulary?

I have a german dataset of financial/annual reports of companies. For example, I use the gensim package to embed my dataset with word2vec or use the huggingface package to embed. How can I find out, ...
Christian01's user avatar
4 votes
2 answers
654 views

Does Llama2 LLMs have real time data?

Meta released their open source LLM, Llama 2. Does Llama 2 have real time data or this was trained on data available till some date?
Rashid's user avatar
  • 169