Adapter Configuration

Classes representing the architectures of adapter modules and fusion layers.

Single Adapter

class transformers.AdapterConfig(original_ln_before: bool, original_ln_after: bool, residual_before_ln: bool, adapter_residual_before_ln: bool, ln_before: bool, ln_after: bool, mh_adapter: bool, output_adapter: bool, non_linearity: str, reduction_factor: int, invertible_adapter: Optional[transformers.adapter_config.InvertibleAdapterConfig] = None, leave_out: List[int] = <factory>)

Base class that models the architecture of an adapter.

classmethod load(config: Union[dict, str], download_kwargs=None, **kwargs)

Loads a given adapter configuration specifier into a full AdapterConfig instance.

Parameters

config (Union[dict, str]) – The configuration to load. Can be either: - a dictionary representing the full config - an identifier string available in ADAPTER_CONFIG_MAP - the path to a file containing a full adapter configuration - an identifier string available in Adapter-Hub

Returns

The resolved adapter configuration dictionary.

Return type

dict

class transformers.PfeifferConfig(original_ln_before: bool = True, original_ln_after: bool = True, residual_before_ln: bool = True, adapter_residual_before_ln: bool = False, ln_before: bool = False, ln_after: bool = False, mh_adapter: bool = False, output_adapter: bool = True, non_linearity: str = 'relu', reduction_factor: int = 16, invertible_adapter: Optional[dict] = InvertibleAdapterConfig(block_type='nice', non_linearity='relu', reduction_factor=2), leave_out: List[int] = <factory>)

The adapter architecture proposed by Pfeiffer et. al., 2020. Described in https://arxiv.org/pdf/2005.00247.pdf.

class transformers.HoulsbyConfig(original_ln_before: bool = False, original_ln_after: bool = True, residual_before_ln: bool = True, adapter_residual_before_ln: bool = False, ln_before: bool = False, ln_after: bool = False, mh_adapter: bool = True, output_adapter: bool = True, non_linearity: str = 'swish', reduction_factor: int = 16, invertible_adapter: Optional[transformers.adapter_config.InvertibleAdapterConfig] = None, leave_out: List[int] = <factory>)

The adapter architecture proposed by Houlsby et. al., 2019. Described in https://arxiv.org/pdf/1902.00751.pdf.

class transformers.InvertibleAdapterConfig(block_type: str, non_linearity: str, reduction_factor: int)

Base class that models the architecture of an invertible adapter module.

Adapter Fusion

class transformers.AdapterFusionConfig(key: bool, query: bool, value: bool, query_before_ln: bool, regularization: bool, residual_before: bool, temperature: bool, value_before_softmax: bool, value_initialized: str)

Base class that models the architecture of an adapter fusion layer.

classmethod load(config: Union[dict, str], **kwargs)

Loads a given adapter fusion configuration specifier into a full AdapterFusionConfig instance.

Parameters

config (Union[dict, str]) – The configuration to load. Can be either: - a dictionary representing the full config - an identifier string available in ADAPTERFUSION_CONFIG_MAP - the path to a file containing a full adapter fusion configuration

Returns

The resolved adapter fusion configuration dictionary.

Return type

dict

class transformers.StaticAdapterFusionConfig(key: bool = True, query: bool = True, value: bool = False, query_before_ln: bool = False, regularization: bool = False, residual_before: bool = False, temperature: bool = False, value_before_softmax: bool = True, value_initialized: str = False)

Static version of adapter fusion without a value matrix. Described in https://arxiv.org/pdf/2005.00247.pdf.

class transformers.DynamicAdapterFusionConfig(key: bool = True, query: bool = True, value: bool = True, query_before_ln: bool = False, regularization: bool = True, residual_before: bool = False, temperature: bool = False, value_before_softmax: bool = True, value_initialized: str = True)

Dynamic version of adapter fusion with a value matrix and regularization. Described in https://arxiv.org/pdf/2005.00247.pdf.