adapters, we support dynamically adding, loading, and deleting of
Embeddings. This section
will give you an overview of these features.
Adding and Deleting Embeddings¶
The methods for handling embeddings are similar to the ones handling adapters. To add new embeddings we call
add_embeddings. This adds new embeddings for the vocabulary of the
In some cases, it might be useful to initialize embeddings of tokens to the ones of another embeddings module. If a
reference_tokenizer are provided all embeddings for tokens that are present in both embeddings are initialized to the embedding provided by the
reference_embedding. The new embedding will be created and set as the active embedding. If you are unsure which embedding
is currently active, the
active_embeddings property contains the currently active embedding.
model.add_embeddings('name', tokenizer, reference_embedding='default', reference_tokenizer=reference_tokenizer)
embedding_name = model.active_embeddings
The original embedding of the transformers model is always available under the name
"default". To set it as the active
embedding simply call the
Similarly, all other embeddings can be set as active by passing their name to the
To delete an embedding that is no longer needed, we can call the
delete_embeddings method with the name of the adapter
we want to delete. However, you cannot delete the default embedding.
Please note, that if the active embedding is deleted the default embedding is set as the active embedding.
Saving and Loading Embeddings¶
You can save the embeddings by calling
save_embeddings('path/to/dir', 'name') and load them with
The path needs to be to a directory in which the weights of the embedding will be saved.
You can also save and load the tokenizer
with the embedding by passing the tokenizer to
model.save_embeddings(path, 'name', tokenizer)
loaded_tokenizer = model.load_embeddings(path, 'name')