Loading Pre-Trained Adapters

Finding pre-trained adapters

AdapterHub.ml provides a central collection of all pre-trained adapters uploaded via Hugging Face’s Model Hub. You can easily find pre-trained adapters for your task of interest along with all relevant information and code snippets to get started.

Note

The original Hub repository (via source="ah") has been archived and migrated to the HuggingFace Model Hub. The Adapters library supports automatic redirecting to the HF Model Hub when attempting to load adapters from the original Hub repository.

Alternatively, list_adapters() provides a programmatical way of accessing all available pre-trained adapters. This will return an AdapterInfo object for each retrieved adapter. E.g., we can use it to retrieve information for all adapters trained for a specific model:

from adapters import list_adapters

adapter_infos = list_adapters(model_name="bert-base-uncased")

for adapter_info in adapter_infos:
    print("Id:", adapter_info.adapter_id)
    print("Model name:", adapter_info.model_name)
    print("Uploaded by:", adapter_info.username)

In case the adapter ID is known, information for a single adapter can also be retrieved via get_adapter_info():

adapter_info = get_adapter_info("AdapterHub/roberta-base-pf-imdb")

print("Id:", adapter_info.adapter_id)
print("Model name:", adapter_info.model_name)
print("Uploaded by:", adapter_info.username)

Using pre-trained adapters in your code

Suppose we have loaded a pre-trained transformer model from Hugging Face, e.g. BERT, and initialized it for adding adapters:

from transformers import BertModel
import adapters

model = BertModel.from_pretrained('bert-base-uncased')
adapters.init(model)

We can now easily load a pre-trained adapter module from Adapter Hub by its identifier using the load_adapter() method:

adapter_name = model.load_adapter('sst-2')

In the minimal case, that’s everything we need to specify to load a pre-trained task adapter for sentiment analysis, trained on the sst-2 dataset using BERT base and a suitable adapter configuration. The name of the adapter is returned by load_adapter(), so we can activate it in the next step:

model.set_active_adapters(adapter_name)

As the second example, let’s have a look at how to load an adapter based on the AdapterInfo returned by the list_adapters() method from above:

from adapters import AutoAdapterModel, list_adapters

adapter_infos = list_adapters()
# Take the first adapter info as an example
adapter_info = adapter_infos[0]

model = AutoAdapterModel.from_pretrained(adapter_info.model_name)
model.load_adapter(adapter_info.adapter_id)

Advanced usage of load_adapter()

To examine what’s happening underneath in a bit more detail, let’s first write out the full method call with all relevant arguments explicitly stated:

model.load_adapter(
    "AdapterHub/roberta-base-pf-imdb",
    version="main",
    load_as="sentiment_adapter",
    set_active=True,
)

We will go through the different arguments and their meaning one by one:

  • The first argument passed to the method specifies the name or path from where to load the adapter. This can be the name of a repository on the HuggingFace Model Hub, a local path or a URL. To get an overview of all available adapters on the Hub, please refer to the Adapter-Hub website.

  • There could be multiple versions of the same adapter available as revisions in a Model Hub repository. To load a specific revision, use the version parameter.

  • By default, the load_adapter() method will add the loaded adapter using the identifier string given as the first argument. To load the adapter using a custom name, we can use the load_as parameter.

  • Finally, set_active will directly activate the loaded adapter for usage in each model forward pass. Otherwise, you have to manually activate the adapter via set_active_adapters().

Saving and loading adapter compositions

In addition to saving and loading individual adapters, you can also save, load and share entire compositions of adapters with a single line of code. Adapters provides three methods for this purpose that work very similar to those for single adapters:

As an example, this is how you would save and load an AdapterFusion setup of three adapters with a prediction head:

# Create an AdapterFusion
model = AutoAdapterModel.from_pretrained("bert-base-uncased")
model.load_adapter("sentiment/sst-2@ukp", config=SeqBnConfig(), with_head=False)
model.load_adapter("nli/multinli@ukp", config=SeqBnConfig(), with_head=False)
model.load_adapter("sts/qqp@ukp", config=SeqBnConfig(), with_head=False)
model.add_adapter_fusion(["sst-2", "mnli", "qqp"])
model.add_classification_head("clf_head")
adapter_setup = Fuse("sst-2", "mnli", "qqp")
head_setup = "clf_head"
model.set_active_adapters(adapter_setup)
model.active_head = head_setup

# Train AdapterFusion ...

# Save
model.save_adapter_setup("checkpoint", adapter_setup, head_setup=head_setup)

# Push to Hub
model.push_adapter_setup_to_hub("<user>/fusion_setup", adapter_setup, head_setup=head_setup)

# Re-load
# model.load_adapter_setup("checkpoint", set_active=True)