Adapters have been introduced as a new alternative of fine-tuning language models on a downstream task. Instead of fine-tuning the full model, a small set of newly introduced task-specific parameters is updated during fine-tuning. The rest of the model is kept fix. Adapters provide advantages in terms of size, modularity and composability while often achieving results on-par with full fine-tuning. We will not go into detail about the theoretical background of adapters in the following but refer to some literature providing more explanations here:

.. note::
*AdapterHub* aims to support a variety adapter setups. To understand our implementation of adapters, there's some important terminology to grasp at first: the difference between *adapter types* and *adapter architectures*.

The *adapter type* describes for which purpose an adapter is used whereas the *adapter architecture* (or *adapter configuration*) describes from which components the adapter modules in the language model are constructed.


The adapter type defines the purpose of an adapter. Currently, all adapters are categorized as one of the following types:

Beginning with version 2, both adapter types are treated identically within the library. The additional invertible adapters are defined via the adapter configuration (see next section). In v1.x, the distinction between task and language adapters was made with the help of the AdapterType enumeration.

.. figure:: img/architecture.png
:width: 350
:align: center

Visualization of possible adapter configurations with corresponding dictionary keys.


The concrete structure of adapter modules and their location in the layers of a Transformer model is specified by a configuration dictionary. This is referred to as the adapter architecture. The currently possible configuration options are visualized in the figure above. When adding new adapters using the add_adapter() method, the configuration can be set providing the config argument. The passed value can be either a plain Python dict containing all keys pictured or a subclass of AdapterConfig.

For convenience, adapter-transformers has some common architectures built-in:

Both of these classes have counterparts with invertible adapters, typically used as language adapters: HoulsbyInvConfig and PfeifferInvConfig.

Furthermore, pre-defined architectures can be loaded from the Hub:

# load "pfeiffer" config from Hub, but replace the reduction factor