AdapterLayerBaseMixin

class transformers.AdapterLayerBaseMixin

An abstract base implementation of adapter integration into a Transformer block. In BERT, subclasses of this module are placed in the BertSelfOutput module and in the BertOutput module.

abstract property adapter_config_key

Gets the name of the key by which this adapter location is identified in the adapter configuration.

adapter_fusion(adapter_setup: transformers.adapters.composition.Fuse, hidden_states, input_tensor, lvl=0)

Performs adapter fusion with the given adapters for the given input.

adapter_parallel(adapter_setup: transformers.adapters.composition.Parallel, hidden_states, input_tensor, lvl=0)

For parallel execution of the adapters on the same input. This means that the input is repeated N times before feeding it to the adapters (where N is the number of adapters).

adapter_split(adapter_setup: transformers.adapters.composition.Split, hidden_states, input_tensor, lvl=0)

Splits the given input between the given adapters.

adapter_stack(adapter_setup: transformers.adapters.composition.Stack, hidden_states, input_tensor, lvl=0)

Forwards the given input through the given stack of adapters.

adapters_forward(hidden_states, input_tensor)

Called for each forward pass through adapters.

add_fusion_layer(adapter_names: Union[List, str])

See BertModel.add_fusion_layer

enable_adapters(adapter_setup: transformers.adapters.composition.AdapterCompositionBlock, unfreeze_adapters: bool, unfreeze_fusion: bool)

Unfreezes a given list of adapters, the adapter fusion layer, or both

Parameters
  • adapter_names – names of adapters to unfreeze (or names of adapters part of the fusion layer to unfreeze)

  • unfreeze_adapters – whether the adapters themselves should be unfreezed

  • unfreeze_fusion – whether the adapter attention layer for the given adapters should be unfreezed

get_adapter_preparams(adapter_config, hidden_states, input_tensor)

Retrieves the hidden_states, query (for Fusion), and residual connection according to the set configuratio

Parameters
  • adapter_config – config file according to what the parameters are passed

  • hidden_states – output of previous layer

  • input_tensor – residual connection before FFN

Returns: hidden_states, query, residual