Model Mixins

These classes provide the basis of adapter module integration into model classes such as adapter saving and loading. Depending on the model, one of these mixins should be implemented by every adapter-supporting model class.

ModelAdaptersMixin

class transformers.ModelAdaptersMixin(config, *args, **kwargs)

Mixin for transformer models adding support for loading/ saving adapters.

abstract add_adapter(adapter_name: str, adapter_type: transformers.adapter_utils.AdapterType, config=None)

Adds a new adapter module of the specified type to the model.

Parameters

  • adapter_name (str) – The name of the adapter module to be added.

  • adapter_type (AdapterType) – The adapter type.

  • config (str or dict or AdapterConfig, optional) – The adapter configuration, can be either: - the string identifier of a pre-defined configuration dictionary - a configuration dictionary specifying the full config - if not given, the default configuration for this adapter type will be used

add_fusion(adapter_names, adapter_fusion_config=None, override_kwargs=None)

Adds AdapterFusion to the model with alll the necessary configurations and weight initializations

Parameters

  • adapter_names – a list of adapter names which should be fused

  • adapter_fusion_config (str or dict) – adapter fusion configuration, can be either: - a string identifying a pre-defined adapter fusion configuration - a dictionary representing the adapter fusion configuration - the path to a file containing the adapter fusion configuration

  • override_kwargs – dictionary items for values which should be overwritten in the default AdapterFusion configuration

freeze_model(freeze=True)

Freezes all weights of the model.

load_adapter(adapter_name_or_path: str, adapter_type: transformers.adapter_utils.AdapterType = None, config: Union[dict, str] = None, version: str = None, model_name: str = None, load_as: str = None, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None, **kwargs) → str

Loads a pre-trained pytorch adapter module from the local file system or a remote location.

Parameters

  • adapter_name_or_path (str) – can be either: - the identifier of a pre-trained task adapter to be loaded from Adapter Hub - a path to a directory containing adapter weights saved using model.saved_adapter() - a URL pointing to a zip folder containing a saved adapter module

  • adapter_type (AdapterType, optional) – The type of adapter to be loaded. If not specified, text_task will be used for adapters loaded from the Hub.

  • config (dict or str, optional) – The requested configuration of the adapter. If not specified, will be either: - the default adapter config for the requested adapter if specified - the global default adapter config

  • version (str, optional) – The version of the adapter to be loaded.

  • model_name (str, optional) – The string identifier of the pre-trained model.

  • load_as (str, optional) – Load the adapter using this name. By default, the name with which the adapter was saved will be used.

Returns

The name with which the adapter was added to the model.

Return type

str

load_adapter_fusion(adapter_fusion_name_or_path: str, load_as: str = None, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None, **kwargs) → str

Loads a pre-trained pytorch adapter module from the local file system or a remote location.

Parameters

  • adapter_fusion_name_or_path (str) – can be either: - the identifier of a pre-trained task adapter fusion module to be loaded from Adapter Hub - a path to a directory containing adapter weights saved using model.saved_adapter() - a URL pointing to a zip folder containing a saved adapter module

  • config (dict or str, optional) – The requested configuration of the adapter fusion. If not specified, will be either: - the default adapter config for the requested adapter fusion if specified - the global default adapter fusion config

  • model_name (str, optional) – The string identifier of the pre-trained model.

  • load_as (str, optional) – Load the adapter using this name. By default, the name with which the adapter was saved will be used.

Returns

The name with which the adapter was added to the model.

Return type

str

save_adapter(save_directory: str, adapter_name: str, meta_dict: dict = None, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None)

Saves an adapter and its configuration file to a directory so that it can be shared or reloaded using load_adapter().

Parameters

  • save_directory (str) – Path to a directory where the adapter should be saved.

  • adapter_name (str) – Name of the adapter to be saved.

Raises

ValueError – If the given adapter name is invalid.

save_adapter_fusion(save_directory: str, adapter_names: list, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None)

Saves an adapter and its configuration file to a directory so that it can be shared or reloaded using load_adapter().

Parameters

  • save_directory (str) – Path to a directory where the adapter should be saved.

  • adapter_name (str) – Name of the adapter to be saved.

Raises

ValueError – If the given adapter name is invalid.

save_all_adapter_fusions(save_directory: str, meta_dict: dict = None, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None)

Saves all adapters of this model together with their configuration to subfolders of the given location.

Parameters

save_directory (str) – Path to a directory where the adapters should be saved.

save_all_adapters(save_directory: str, meta_dict: dict = None, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None)

Saves all adapters of this model together with their configuration to subfolders of the given location.

Parameters

save_directory (str) – Path to a directory where the adapters should be saved.

set_active_adapters(adapter_names: list)

Sets the adapter modules to be used by default in every forward pass. This setting can be overriden by passing the adapter_names parameter in the foward() pass. If no adapter with the given name is found, no module of the respective type will be activated.

Parameters

adapter_names (list) – The list of adapters to be activated by default. Can be a fusion or stacking configuration.

set_adapter_config(adapter_type: transformers.adapter_utils.AdapterType, adapter_config)

Sets the adapter configuration of the specified adapter type.

Parameters

  • adapter_type (AdapterType) – The adapter type.

  • adapter_config (str or dict) – adapter configuration, can be either: - a string identifying a pre-defined adapter configuration - a dictionary representing the adapter configuration - the path to a file containing the adapter configuration

set_adapter_fusion_config(adapter_fusion_config, override_kwargs=None)

Sets the adapter fusion configuration.

Parameters

adapter_fusion_config (str or dict) – adapter fusion configuration, can be either: - a string identifying a pre-defined adapter fusion configuration - a dictionary representing the adapter fusion configuration - the path to a file containing the adapter fusion configuration

abstract train_adapter(adapter_names: list)

Sets the model into mode for training the given adapters.

abstract train_fusion(adapter_names: list)

Sets the model into mode for training of adapter fusion determined by a list of adapter names.

ModelWithHeadsAdaptersMixin

class transformers.ModelWithHeadsAdaptersMixin(config, *args, **kwargs)

Mixin adding support for loading/ saving adapters to transformer models with head(s).

add_adapter(adapter_name: str, adapter_type: transformers.adapter_utils.AdapterType, config=None)

Adds a new adapter module of the specified type to the model.

Parameters

  • adapter_name (str) – The name of the adapter module to be added.

  • adapter_type (AdapterType) – The adapter type.

  • config (str or dict, optional) – The adapter configuration, can be either: - the string identifier of a pre-defined configuration dictionary - a configuration dictionary specifying the full config - if not given, the default configuration for this adapter type will be used

load_adapter(adapter_name_or_path: str, adapter_type: transformers.adapter_utils.AdapterType = None, config: Union[dict, str] = None, version: str = None, model_name: str = None, load_as: str = None, with_head: bool = True, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None, **kwargs) → str

Loads a pre-trained pytorch adapter module from the local file system or a remote location.

Parameters

  • adapter_name_or_path (str) – can be either: - the identifier of a pre-trained task adapter to be loaded from Adapter Hub - a path to a directory containing adapter weights saved using model.saved_adapter() - a URL pointing to a zip folder containing a saved adapter module

  • adapter_type (AdapterType, optional) – The type of adapter to be loaded. If not specified, text_task will be used for adapters loaded from the Hub.

  • config (dict or str, optional) – The requested configuration of the adapter. If not specified, will be either: - the default adapter config for the requested adapter if specified - the global default adapter config

  • version (str, optional) – The version of the adapter to be loaded.

  • model_name (str, optional) – The string identifier of the pre-trained model.

  • load_as (str, optional) – Load the adapter using this name. By default, the name with which the adapter was saved will be used.

Returns

The name with which the adapter was added to the model.

Return type

str

save_adapter(save_directory: str, adapter_name: str, with_head: bool = True, meta_dict: dict = None, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None)

Saves an adapter and its configuration file to a directory so that it can be shared or reloaded using load_adapter().

Parameters

  • save_directory (str) – Path to a directory where the adapter should be saved.

  • adapter_name (str) – Name of the adapter to be saved.

Raises

ValueError – If the given adapter name is invalid.

save_all_adapters(save_directory: str, with_head: bool = True, meta_dict: dict = None, custom_weights_loaders: Optional[List[transformers.adapter_model_mixin.WeightsLoader]] = None)

Saves all adapters of this model together with their configuration to subfolders of the given location.

Parameters

save_directory (str) – Path to a directory where the adapters should be saved.

train_adapter(adapter_names: list)

Sets the model into mode for training the given adapters.

train_fusion(adapter_names: list)

Sets the model in mode for training of adapter fusion determined by a list of adapter names.