Embeddings: How can I find out, that I'm out-of-vocabulary?

Question

I have a german dataset of financial/annual reports of companies. For example, I use the gensim package to embed my dataset with word2vec or use the huggingface package to embed.

How can I find out, that I'm out-of-vocabulary? I want to check it, because - first - it's a german dataset and - second - in a specific domain.

Are there any function or methods in the packages to check that or do I have to code workaround?

Answer 1

It's going to depend on the library. For gensim, the vocab is an attribute of the word2vec model (see this answer: https://datascience.stackexchange.com/a/56106/151809).

For models that use byte-pair tokenization, you won't have out-of-vocabulary tokens as individual bytes are part of the vocabulary, and any input can be encoded as bytes.

Models, like BERT, that use subword tokenization can have OOV tokens, as the vocabulary still consists of a finite number of strings. In huggingface, you can look for an unk_token attribute in tokenizer objects. If your word gets encoded as an unk_token, then it is out of vocabulary.

Answer 2

How can I find out, that I'm out-of-vocabulary?

1. See if the embedding model is word-based or uses subword embeddings (e.g., Bert uses WordPieces).
2. If word-based, there should be a way to list the vocabulary. Alternatively, one can try comparing the embedding of a given word against the embedding of some made-up word (if it's the same embedding, then the given word is out-of-vocabulary).