Questions tagged [transformers]

Transformer models are a type of deep learning model that typically involve encoder / decoder architecture with self-attention. Use this tag to ask questions about transformer model fundamentals, architecture, implementation or training.

Filter by
Sorted by
Tagged with
-2 votes
1 answer
26 views

What is the best transformer to generate text based on a dataset?

I'm working with huggingface to run TensorFlow models locally. I'm using this as a learning exercise since I've never worked with AI before (I'm more interested in learning the process than the actual ...
PedroC88's user avatar
0 votes
1 answer
83 views

Architectures of LLMs

What does the architecture of a LLM look like? Which part stores the enormous parameters? I know the transformer model, but I don’t know how to scale it sensibly as LLMs do.
Zirui Wang's user avatar
0 votes
0 answers
39 views

How to populate labels for dataset preprocessing before LLM fine tuning?

I am using vicgalle/alpaca-gpt4 dataset to fine tune a LLM. I am trying to figure out how to tokenize the dataset's output column into labels within a dataset preprocessor. Please see the below ...
Intrastellar Explorer's user avatar
0 votes
0 answers
14 views

how to load distcp checkpoint files?

I have fintuned full aparmeters of mistral 7-b model, and i have used FDSP in HF accelerate I have a checkpoint which is place in a folder pytorch_model_0, which contains multiple distcp files. how ...
Mustafa Alahmid's user avatar
0 votes
0 answers
13 views

Interpretation of Evaluation Values of Augmented SBERT Training with EmbeddingSimilarityEvaluator()

I train a BI-Encoder to get an Augmented SBERT and I get a final training result. How can I interpret the following output of the final training result? EmbeddingSimilarityEvaluator: Evaluating the ...
Christian01's user avatar
0 votes
0 answers
24 views

The using of golden dataset in Augmented SBERT Training

I use the training strategy of Augmented SBERT (Domain-Transfer) to train an augmented SBERT. In the code example they use the golden-dataset (STSb) for the training evaluator. Here two code snippets ...
Christian01's user avatar
1 vote
0 answers
42 views

Are LLMs the quickest, most cost-effective way to assess affinity between two texts?

I would like to compare thousands of text descriptions with a specific 'master' input and receive boolean (true/false) results for each comparison, based on their affinity. The following is an example ...
Nic G's user avatar
  • 19
1 vote
0 answers
34 views

Get Llama-2 Rotary Embeddings

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the rotary embeddings and how can I ...
Christian01's user avatar
0 votes
0 answers
37 views

Reversible Transformer and LSH vs. SOA Models

In the DeepLearning.ai specialization on Natural Language Processing, the instructor claims that the Reformer, using LSH and reversible layers, is able to ingest context lengths of 1M+ tokens and ...
mrplants's user avatar
  • 101
0 votes
1 answer
67 views

Is the simulators viewpoint still valid?

When considering the behavior of large language models, there is not yet a single canonical framework for interpreting their output in context. Common proposed frameworks include: Agents: The LLM ...
Corbin's user avatar
  • 111
11 votes
3 answers
2k views

How do I "teach" a large language model new knowledge?

Suppose I have a copy of a pre-trained transformer-based large language model like Google's T5 or Meta's Llama. Due to the pre-training, it contains a lot of knowledge. However, I want to teach the ...
Ian Campbell's user avatar