An abstract base implementation of adapter integration into a Transformer block. In BERT, subclasses of this module are placed in the BertSelfOutput module and in the BertOutput module.
Gets the name of the key by which this adapter location is identified in the adapter configuration.
adapter_fusion(adapter_setup: transformers.adapters.composition.Fuse, hidden_states, input_tensor, lvl=0)¶
Performs adapter fusion with the given adapters for the given input.
adapter_parallel(adapter_setup: transformers.adapters.composition.Parallel, hidden_states, input_tensor, lvl=0)¶
For parallel execution of the adapters on the same input. This means that the input is repeated N times before feeding it to the adapters (where N is the number of adapters).
adapter_split(adapter_setup: transformers.adapters.composition.Split, hidden_states, input_tensor, lvl=0)¶
Splits the given input between the given adapters.
adapter_stack(adapter_setup: transformers.adapters.composition.Stack, hidden_states, input_tensor, lvl=0)¶
Forwards the given input through the given stack of adapters.
Called for each forward pass through adapters.
add_fusion_layer(adapter_names: Union[List, str])¶
enable_adapters(adapter_setup: transformers.adapters.composition.AdapterCompositionBlock, unfreeze_adapters: bool, unfreeze_fusion: bool)¶
Unfreezes a given list of adapters, the adapter fusion layer, or both
adapter_names – names of adapters to unfreeze (or names of adapters part of the fusion layer to unfreeze)
unfreeze_adapters – whether the adapters themselves should be unfreezed
unfreeze_fusion – whether the adapter attention layer for the given adapters should be unfreezed
get_adapter_preparams(adapter_config, hidden_states, input_tensor)¶
Retrieves the hidden_states, query (for Fusion), and residual connection according to the set configuratio
adapter_config – config file according to what the parameters are passed
hidden_states – output of previous layer
input_tensor – residual connection before FFN
Returns: hidden_states, query, residual
- abstract property