8

Can embeddings from one LLM's reused by others - any specific standardization exists on embeddings representation? reference - https://platform.openai.com/docs/guides/embeddings/what-are-embeddings

5
  • It looks like this question needs more work. Please remember the standard SO Inc. public Q&A norms set that posts should be self-contained. This post also doesn't show the tone and high quality that will help make this community attractive to subject matter experts.
    – Wicket
    Jul 28, 2023 at 1:42
  • 2
    @Wicket For the purpose of community building, would it be possible to please not leave so many of these negative comments? They're going to discourage new users from using the site. Please leave constructive comments, describing concrete improvements, instead. Jul 28, 2023 at 2:07
  • @RebeccaJ.Stones Why do you qualify my comment as "negative"? How do you suggest providing timely feedback that encourages new users to post high-quality questions?
    – Wicket
    Jul 28, 2023 at 2:09
  • 1
    @Wicket Your comment points out problems, but don't explain how to resolve any of them. Some of these criticisms are vague, to the point of being completely unactionable (e.g. “doesn't show the tone and high quality”) – the majority of your comment is just negativity for negativity's sake. Phrasing it something like “questions are expected to be self-contained: please include the relevant information” – or the equivalent for whatever point you were trying to make about “tone” – would be much more helpful.
    – wizzwizz4
    Jul 28, 2023 at 14:36
  • @wizzwizz4 Thanks for your detailed feedback. Please take a look to genai.meta.stackexchange.com/q/110/12
    – Wicket
    Jul 28, 2023 at 16:25

2 Answers 2

4

Embeddings are not universal, they vary per framework/API.

Read an intro here: https://gpt-index.readthedocs.io/en/latest/core_modules/model_modules/embeddings/root.html

And then see various embeddings here:

If you index documents in a vector database with the purpose of providing docs to the ChatGPT Retrieval plugin, then you'll want to use the OpenAI Embeddings.

4
  • 1
    @Wicket: there's nothing wrong with this question! Jul 28, 2023 at 8:24
  • @Wicket or @RebeccaJ.Stones, can you add tag embeddings? I cannot yet add tags Jul 28, 2023 at 8:25
  • Welcome! @ mentions doesn't work as they do on Twitter. They only send a notification to post participants. In this case Rebecca and I didn't got an @ mention notification because we didn't comment or edit this answer.
    – Wicket
    Jul 28, 2023 at 12:46
  • I respectfully disagree with "there 's nothing wrong with this question". I plan to share details about what is wrong and how to improve the question on GenAI Meta later. When I do so, I will include a link in a comment.
    – Wicket
    Jul 28, 2023 at 16:31
0

There is no standardization for embedding vectors across different Large Language Models (LLMs). Each LLM trains its own embeddings as part of the model architecture and training process. As a result, embedding vectors differ between models for several reasons:

  • Different model architectures: LLMs such as GPT-3, Bert, PaLM, etc. have different underlying architectures that affect how embeddings are learned. For example, some may use transformer blocks, while others may use convolutional networks.
  • Different training data: The text data used to train the models also affects the embedding space. Models trained on more data from different sources will develop richer embeddings.
  • Different preprocessing: Things like tokenization and vocabulary creation differ between models and affect the embeddings.
  • Random initialization: Embeddings start with random initialization and evolve through training. The randomness introduces variation.
  • Stochastic training: Training algorithms rely on randomness, which leads to variation in the final embeddings.
  • Different objectives: Models optimize different loss functions and objectives, which affect how embeddings are updated during training.

In short, embeddings capture semantic meaning, but they are not standardized across models. Each LLM learns embeddings tailored to its own architecture, data, and training process. The embeddings are therefore not directly transferable or reusable between different LLMs without modification.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.