1

In my project I follow the retrieval augmented generation (RAG) approach. I want to create embeddings for my own dataset and use it in combination with llama-2. In the dataset are german annual reports, 548 reports as pdf-files with about 300 sites per report. Next, I want to load the data in a vector store, but first I think I have to create the embeddings.

And now, there are serveral questions and I need some best-practice:

  1. Do I have to train my own embeddings model or can I use models like word2vec of the gensim package or a pretrained model like BERT and take the hidden state?

  2. Can I use any embeddings model? I think they train on a specific corpus and if my words aren't in the training corpus, I will get bad result or what do you think?

I hope you can grap me under the arms and help me to get a better understanding.

1 Answer 1

2

Do I have to train my own embeddings model or can I use models like word2vec of the gensim package or a pretrained model like BERT and take the hidden state?

One can use pretrained models. sentence-transformers has more recent models.

Can I use any embeddings model?

The pretrained model needs to support the language of your document, e.g. https://www.sbert.net/docs/pretrained_models.html?highlight=german

if my words aren't in the training corpus, I will get bad result or what do you think?

If out-of-vocabulary words are an issue, one can finetune the pretrained models.

2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.